Gen 5 — Reinforcement Learning (State of the Art)

# Gen 5 — Reinforcement Learning

> **Current state, 2026-07-17.** Imitation learning with G48 teacher. Data collection in progress.

## Feature Vector (290 dimensions)

| Category | Dims | Fields |
|---|---|---|
| Card/Equity | 20 | hs, ppot, npot, nutpot, rpot + hand category (9 one-hot) + draws (6) |
| Range-weighted equity | 5 | hs_rw, ppot_rw, npot_rw, nutpot_rw, win_prob_rw — exact enumeration vs tightest opponent, weighted by HandRange |
| Multiway MC equity | 5 | hs_mc, ppot_mc, npot_mc, nutpot_mc, win_prob_mc — 1000-sample MC vs all opponents |
| Board texture | 10 | board_count, wetness, suit_count, connectivity, high_card |
| Betting/Situation | 25 | pot_bb, to_call_bb, spr, stack_bb + position (9) + street (4) + counts |
| Hero state | 5 | wagered_bb, is_aggressor, is_pfr_aggressor, facing_bet |
| Opponent models (×3) | 60 | 20 fields each from Gen 4 PlayerStats |
| Action history (×8) | 160 | Per-action: player_offset, street, action_type, amount_bb, pot_after_bb |
| **Total** | **290** | |

## Action Space (11 discrete → 3+5 multi-head)

Underlying: `Fold, Check, Call, BetThird(0.33), BetHalf(0.50), BetTwoThird(0.67), BetPot(1.0), Bet2x(2.0), Raise25x(2.5×), RaisePot(pot), AllIn`

Multi-head mapping: action_type (passive/call/aggressive) + sizing_class (third/half/2third/pot+/allin).

## Training Pipeline

### Phase 1: Imitation Learning (GPU, CUDA)
1. Collect transitions from teacher bot in simulation
2. Train multi-head MLP with class-weighted cross-entropy + sizing-weighted MSE
3. Feature normalization (per-feature standardization saved to `.norm.json`)

**Latest model**: `postflop_v13` — trained on G48 teacher data.

### Data Collection Status (2026-07-17)
- **Short-handed (3-6 handed)**: ✅ COMPLETE — 24.1M transitions from 10M hands (39GB)
- **Full-ring (7-8 handed)**: 🔄 RUNNING — ~3.5M of ~20M transitions (7 opponent types: chump, flock, gen1, gen2, gen3, loose, tight)
- **Teacher**: G48 (`teacher_postflop = "formula"`, 10K MC samples)
- **Config**: `configs/bots/gen5_collect_g48.toml`

### Phase 2: RL Fine-Tuning (REINFORCE) — PLANNED
- Frozen backbone + RL-adapted heads only
- Mixed table: 1 RL + mixed opponents
- Quick eval every 50 iterations, deep eval every 200

### Phase 3: Live Fine-Tuning (future)
- Collect Torn session data, fine-tune on real opponent distributions

## Next Steps
1. Wait for full-ring collection → merge all transitions
2. Train `postflop_v14` on combined ~44M transitions
3. Evaluate v14 vs G48 in simulation
4. PPO fine-tuning if imitation plateaus
5. Deploy via `gen5_rl.toml`

## Key Technical Details

### Candle LayerNorm Bug (FIXED)
Candle-nn 0.8.4's `ops::layer_norm()` has no backward pass. Replaced with manual layer_norm in `multi_head.rs`, `network.rs`, `range_net.rs`.

### Equity Feature Sources
- **Raw equity** (features 0-4): `HandPotential::compute()` — exact enumeration, uniform opponent
- **Range-weighted equity** (features 20-24): `compute_vs_range()` — exact enumeration weighted by tightest opponent's HandRange
- **Multiway MC equity** (features 25-29): `compute_vs_range_multiway_shared()` — 1000 samples

## Key Files

| File | Content |
|---|---|
| `holdem_bots/src/gen5/features.rs` | 290-dim feature extraction + range-weighted + multiway equity |
| `holdem_bots/src/gen5/multi_head.rs` | Multi-head model + trainer (LayerNorm fix, REINFORCE) |
| `holdem_bots/src/gen5/rl_strategy.rs` | RL strategy + `compute_range_equity()` + `teacher_postflop` config |
| `holdem_bots/src/gen5/network.rs` | Candle MLP policy + `best_device()` GPU selection |
| `holdem_bots/src/gen5/recorder.rs` | JSONL transition recorder |
| `holdem_bots/src/gen5/actions.rs` | Action space + masking + `action_to_heads()` |
| `holdem_bots/src/bin/train_gen5_multi.rs` | Imitation trainer (GPU/CUDA) |
| `holdem_bots/src/bin/train_gen5_rl.rs` | RL trainer (REINFORCE) |
| `scripts/gen5_collect_g48.sh` | Short-handed collection (3-6 handed) |
| `scripts/gen5_collect_g48_fullring.sh` | Full-ring collection (7-8 handed) |
| `models/postflop_v13.safetensors` | Current imitation model |
| `models/rl_current.safetensors` | RL model (latest) |

## Design Decisions

- **NN takes equity + opponent stats as input** (not range prediction NN) — pragmatic, builds on Gen 4
- **Unified network** for preflop + postflop (preflop zeroes board features)
- **Opponent features as input** — cold start = all-zeros, network adapts as stats accumulate
- **Hybrid training**: offline imitation → offline RL → live fine-tuning (live play too slow for weight updates)
- **Curriculum plan**: Beat Gen 1 → 2 → 3 → 4 → self-play convergence (Nash approximation)


id: c7a39e16770542a08004fd73e971ca1f
parent_id: 5a06903f7db44bfcb4c8c8a9cf0d2326
created_time: 2026-06-28T05:17:26.889Z
updated_time: 2026-07-17T08:42:06.844Z
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 1782623846889
user_created_time: 2026-06-28T05:17:26.889Z
user_updated_time: 2026-07-17T08:42:06.844Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
is_shared: 0
share_id: 
conflict_original_id: 
master_key_id: 
user_data: 
deleted_time: 0
type_: 1