Roadmap

# Roadmap

> Updated 2026-07-17 (rev 5).

---

## Phase 1: Simulation Testbed ✅ COMPLETE

- Simulation engine (`holdem_core`) — cash games, SNG, MTT
- Strategy framework — trait-based composition, **22 registered strategies**
- Gen 1 strategies — static rules
- Configurable thresholds — **80+ PostflopThresholds fields** with TOML overrides
- Experiment GUI (`holdem_gui`) — web-based parameter sweep tool (84 params)
- `hand_replay` crate — PokerStars parser + bot decision replay/regression
- **PEM2 preflop equity matrix** — 169×169 distribution matrix (10 metrics × 8 bins)

---

## Phase 2: Gen 2-4.8 Strategy Development ✅ COMPLETE (in production)

- ✅ Gen 2: Monte Carlo EHS equity engine + KEHU Nash HU + NG preflop
- ✅ Gen 3: Range-aware equity (single predicted opponent range)
- ✅ Gen 4: Adaptive opponent modelling (EMA observer, three-tier blend, range narrowing)
- ✅ Gen 4.5: Playability-enhanced preflop + adaptive postflop
- ✅ Gen 4.7: Matrix-based equity preflop (PEM2)
- ✅ **Gen 4.8: Formula-based postflop with observer ranges — CURRENT PRODUCTION POSTFLOP**
  - Unified score: `value_score + ppot_w×ppot - npot_w×npot`
  - `value_score = hs×(1 - npot_w×npot) + nutredraw_w×nutpot×(1-hs) - behind_penalty_w×(1-hs)×(1-ppot)`
  - call_threshold with multiway pot-odds cap: `0.28/√opp` (non-river), `0.15/√opp` (river)
  - Bluff gating on weak opponents, stack-off AllIn guard, position-aware sizing
  - Observer-based range equity (shared with G45 via `build_ranges_and_equity()`)
  - Cold-start priors: VPIP 0.30, PFR 0.15, AF 1.0, fold_to_bet 0.50
  - MC budget: 40,000 samples (divided by opponent count, floor 8K)
  - **Config: `cash_nl_g48_live.toml`**

### G47 Preflop — CURRENT PRODUCTION PREFLOP
- PEM2 matrix equity (zero MC — pure lookup)
- Unified score: `equity + ppot_w×ppot + nutpot_w×nutpot - npot_w×npot`
- Position-based thresholds, stack-depth sigmoid
- Per-opponent PFR/VPIP for reraise/call paths (with table-level fallback)
- **Config: `cash_nl_g47_live.toml`**

### Recent Bug Fixes (2026-07-16)
- ✅ Fixed `remaining_stack_share` double-counting bug (formula_postflop:887)
- ✅ Relaxed multiway call margin 0.35→0.28 for non-river streets
- ✅ Postflop fold logging upgraded to info level
- ✅ PlayerId seat mismatch fix (bot_id tracking through seat changes)
- ✅ Call amount reconciliation (downward + upward current_bet correction)
- ✅ Blinds synthesis for missing blind-posting events
- ✅ G47 call-path sizing narrowing (`call_range` narrows by bet sizing)
- ✅ G48 KK overplay on Ace-high boards — board-hit boost (narrow_board_hit_boost=3.0)
- ✅ Per-player looseness (`aggressor_vpip()` keys off bettor's individual VPIP)
- ✅ Per-opponent PFR in G47 reraise path (was table-level)
- ✅ RaiseBelowMinimum fix (BB rounding clamp in raise sizing)
- ✅ Draw-heavy board trap guard (`board_is_drawy()` in gen3 + formula_postflop)
- ✅ Deep-stack overbet allin fix (raise to opponent effective stack, not full shove)
- ✅ Behind-penalty in value_score (behind_penalty_weight=0.15)

### Open Tuning Items
- [x] ~~Postflop equity matrix~~ — DEFERRED (MC rollouts are fine for live play)
- [x] All 2026-07-16 tuning items complete (see above)
- [x] Chump_bot RaiseBelowMinimum — DONE (BB rounding clamp)

---

## Phase 3: Live Integration (Torn) ✅ FUNCTIONAL

### Completed
- Event-stream protocol (`poker_protocol::TableEvent`)
- `live_server` (Axum gateway) — per-table state machine, bot auto-detection
- Torn TamperMonkey scraper v0.1.11 — DOM parsing (seats, stacks, cards, actions, blinds)
- Profile persistence (player ID keyed, survives restarts)
- Graceful shutdown/restart endpoints
- **Default bot: G48 live (primary) + G47 live (fallback)**
- Per-hand event/decision logging in `/tmp/super_marvin_hands/`
- ✅ **DOM snapshot mode** — `dom_snapshot` on `waiting_for_action` + `raw_text` on `player_action` events. Userscript captures table HTML (gated by `window.__marvinDomDebug = true`). Server logs snapshots to per-hand decision logs.
- ✅ Raise amount regex fix (k/M suffix handling in parser.ts)
- ✅ Corrupt blinds false positive fix (silent when sb/bb non-zero but not yet seen)
- ✅ Raise tolerance fix (1% → 5% for Torn chip denomination rounding)

### Open Issues
- [ ] Multi-table support
- [ ] Native desktop scrapers (deferred)

---

## Phase 4: Gen 5 Reinforcement Learning 🔄 IN PROGRESS

> Gen5 data collection RESUMED with G48 teacher (2026-07-17).

### Implemented
- **290-dim** feature extraction (equity, board, betting, opponents, actions, range-weighted + multiway)
- 11-discrete-action space with legality masking
- JSONL transition recorder (thread-safe, bounded)
- Candle MLP policy network (290→512→LN→512→LN→256→3+5+1 multi-head)
- Training CLI with GPU/CUDA, class-weighted loss, focal loss, gradient clipping
- Multi-head policy (action + sizing heads, class weights)
- DAgger strategy (teacher fallback + model inference)
- PPO self-play loop (v11 = best deployed model, +27.9 avg chips/hand)
- ✅ **G48 teacher wiring** — `teacher_postflop` config field: `"formula"` → `FormulaPostflopGen48`, `"adaptive"` → `AdaptivePostflopGen4`

### Data Collection (2026-07-17)
- **Short-handed batch (3-6 handed)**: COMPLETE — 24.1M transitions from 10M hands (39GB)
- **Full-ring batch (7-8 handed)**: RUNNING — 8 tables × 1.25M hands, ETA ~23h
- Combined target: ~44M transitions covering full Torn seat distribution
- Teacher: G47 preflop (PEM2) + G48 postflop (formula, 10K MC budget for collection speed)
- Config: `gen5_collect_g48.toml` (mirrors `cash_nl_g48_live.toml`)
- Scripts: `gen5_collect_g48.sh` (short-handed), `gen5_collect_g48_fullring.sh` (full-ring)

### Models
| Version | Training | Eval Acc | Status |
|---------|----------|----------|--------|
| v1-v4 | Various | Degenerate | Fixed (LayerNorm, action mapping, features) |
| v5-v10 | Imitation (G4.5 teacher) | 45-46% | Production baseline |
| v11 | 3.15M transitions, 290-dim, GPU | 45.2% | Previous best (+27.9 chips/hand) |
| v12-v13 | G47 teacher, class weights | 97.6% action, 76.6% sizing | Plateaued — teacher quality ceiling |
| **v14** | **G48 teacher, ~44M transitions** | **TBD** | **Next: train after collection** |

### Next Steps
1. ✅ Wire G48 as teacher (`teacher_postflop = "formula"`)
2. ✅ Collect short-handed data (24.1M transitions)
3. ⏳ Collect full-ring data (~20M transitions, running)
4. ⬜ Train v14 on merged G48 teacher transitions
5. ⬜ Evaluate v14 vs G48 in simulation
6. ⬜ PPO fine-tuning (if imitation plateaus again)

---

## Key Metrics (2026-07-17)

| Metric | Value |
|--------|-------|
| Registered strategies | 22 |
| Production preflop | **G47** (`g47_equity_preflop` — PEM2 matrix) |
| Production postflop | **G48** (`g48_formula_postflop` — formula + observer ranges) |
| Live integration | Torn — functional, userscript v0.1.11 |
| MC budget (live) | 40,000 (divided by opponents, floor 8K) |
| PEM2 matrix | 8.7 MB, 10 metrics × 8 bins × 169² pairs |
| Regression tests | 49 hand replay tests + 478 lib tests (all pass) |
| Gen 5 transitions | 24.1M (short-handed done) + ~20M (full-ring in progress) |
| Live sessions | 2026-07-08 (~42.5M winnings), 2026-07-16 (clean, no issues) |

---

## Immediate Next Session Priorities

1. **Wait for full-ring collection** (~23h, auto-running) → merge both batches
2. **Train v14** — `train_gen5_multi <merged> 30 1024 1e-4 models/postflop_v14.safetensors --street postflop` (needs `--features gen5_nn`)
3. **Evaluate v14** vs G48 in simulation — target: beat G48 or match with lower latency
4. **PPO fine-tuning** if v14 imitation plateaus (REINFORCE/PPO on RL-sampled transitions)
5. **Live monitoring** — keep watching for reconciliation warnings, test DOM snapshot mode
6. **Update `gen5_rl.toml`** deploy config to point at v14 multi-head model + norm.json


id: 47031623e2c6451fb36730ffe93380ea
parent_id: 2c8da247905946c3aa19eb4936e16323
created_time: 2026-05-31T10:37:02.831Z
updated_time: 2026-07-17T04:42:35.285Z
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 1780223822831
user_created_time: 2026-05-31T10:37:02.831Z
user_updated_time: 2026-07-17T04:42:35.285Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
is_shared: 0
share_id: 
conflict_original_id: 
master_key_id: 
user_data: 
deleted_time: 0
type_: 1