Range Prediction Accuracy Tracking — Implementation & Tournament Results

# Range Prediction Accuracy Tracking — Implementation & Tournament Results

## Date: 2026-06-25

## What Was Built

### Showdown-based accuracy tracking
- Added `showdown_event` to the `Strategy` trait (default no-op), forwarded through `ModularStrategy` and `StrategyBot`
- Gen4's `AdaptivePostflopGen4` implements `showdown_event` → routes revealed cards to `Observer::on_showdown`
- Observer stores per-opponent `HandRange` predictions at decision time, checks them against actual cards at showdown
- Thread-local `AccuracyData` collector accumulates records across all Gen4 bots in the simulation
- Summary printed after simulation in `main.rs`

### Metrics tracked
- **Hit rate**: fraction of shown hands with weight > 0 in predicted range
- **Average weight**: mean predicted weight of actual hands (higher = better prediction)
- **Weight distribution**: high (≥0.8) / mid (0.3-0.8) / low (0-0.3) / zero
- **By confidence**: low (<0.3) / mid (0.3-0.7) / high (≥0.7) observer confidence
- **By opponent VPIP**: tight (<0.25) / medium (0.25-0.45) / loose (≥0.45)
- **By street**: flop / turn / river

### Files modified
- `holdem_bots/src/common/modular.rs` — Strategy trait + ModularStrategy showdown forwarding
- `holdem_bots/src/strategy_bot.rs` — Forward showdown_event to strategy
- `holdem_bots/src/gen4/observer.rs` — Prediction storage + accuracy checking in on_showdown
- `holdem_bots/src/gen4/adaptive.rs` — Store predictions during get_action, implement showdown_event
- `holdem_bots/src/gen4/accuracy.rs` (new) — Thread-local collector + summary reporting
- `holdem_bots/src/gen4/mod.rs` — Export accuracy module
- `holdem_bots/src/main.rs` — Print accuracy summary after cash simulation
- `scripts/range_eval.sh` (new) — Overnight evaluation script

## Tournament Results (10 seeds × 100k hands, old binary)

| Bot Type | Avg Profit/Hand | BB/100 | StdDev |
|----------|-----------------|--------|--------|
| Chump    | -235.25 | -1176.2 | 2.44 |
| Flock    | +55.96 | +279.8 | 1.47 |
| Gen2     | +39.27 | +196.3 | 1.08 |
| Gen3     | +60.06 | +300.3 | 0.95 |
| **Gen4** | **+79.95** | **+399.8** | **1.63** |

**Gen4 vs Gen3: +99.5 BB/100 improvement** (33% relative). Range prediction is clearly valuable.

## Smoke Test Accuracy Data (500 hands, chump opponents only)

```
Total showdowns checked: 768
In-range hit rate: 100.0% (768/768)
Average predicted weight: 0.624

Weight distribution: high(≥0.8)=444 (58%) | mid=8 (1%) | low=316 (41%) | zero=0

By observer confidence:
  Low  (<0.3):  100.0% / 0.549 / 70
  Mid  (0.3-0.7): 100.0% / 0.613 / 109
  High (≥0.7): 100.0% / 0.635 / 589
```

**Observation**: avg_weight increases with confidence (0.549 → 0.635), showing adaptation works.

## Overnight Run

Launched `scripts/range_eval.sh`: 10 seeds × 50k hands with accuracy tracking.
Full tournament lineup (2×Chump + 2×Flock + 2×Gen2 + 2×Gen3 + 2×Gen4).

Expected: accuracy data across different opponent types (tight Flock vs loose Chump vs medium Gen2/Gen3).

## Next Steps
- Analyze overnight accuracy data: are ranges predicted well for different opponent types?
- Tune range parameters based on accuracy findings
- Consider gradient range tails (instead of hard cutoff) for better prediction

id: 226d902e8ddd4baebd546b6f2fe2613b
parent_id: abc167de888d41bf9391c373e90dec8d
created_time: 2026-06-25T19:17:48.086Z
updated_time: 2026-06-30T05:34:40.215Z
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 1782415068086
user_created_time: 2026-06-25T19:17:48.086Z
user_updated_time: 2026-06-25T19:17:48.086Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
is_shared: 0
share_id: 
conflict_original_id: 
master_key_id: 
user_data: 
deleted_time: 1782797680215
type_: 1