poe2-bot/tools/python-detect

YOLO Detection Pipeline

Capture frames → annotate → train → detect live in-game.

Setup

cd tools/python-detect
python -m venv .venv
.venv/Scripts/activate        # Windows
pip install ultralytics opencv-python
# GPU training (NVIDIA):
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128 --force-reinstall

1. Capture Frames

In the app Debug tab, click Burst Capture during a boss fight. Frames save to training-data/raw/ every 200ms. Click Stop Capture when done.

2. Annotate

python annotate.py                              # defaults to ../../training-data/raw
python annotate.py ../../training-data/raw

tools\python-detect\.venv\Scripts\python.exe tools\python-detect\annotate.py training-data\raw

Toolbar

Clickable buttons at the top of the window:

Button	Hotkey	Action
Predict	P	Run YOLO model on current image to auto-generate boxes
Save+Next	Space	Save labels and advance to next image
Filter	F	Cycle filter: All → Unlabeled → Labeled → per-class
Undo	Z	Undo last change
Del Image	X	Delete current image file and its label

Controls

Action	Input
Draw new box	Left-drag on empty area
Select box	Left-click on box
Move box	Left-drag box body
Resize box	Left-drag corner handle
Cycle class	Right-click on box
Set class	1-9 keys (selected box or new-box default)
Delete box	Delete / Backspace
Navigate	Left / Right arrow
Deselect	E
Quit	Q / Escape (auto-saves)

Predict (P)

Press P or click the Predict button to run the current YOLO model (models/boss-v1.pt) on the image. It replaces existing boxes with model predictions (undoable with Z). The model lazy-loads on first use.

Workflow: press P to auto-predict → adjust/delete bad boxes → Space to save+next.

Classes

Edit the CLASSES list at the top of annotate.py to add new boss types:

CLASSES = ["kulemak", "arbiter"]

3. Split Dataset

Create a train/valid split (85/15) for training. Example structure:

training-data/boss-dataset/
├── data.yaml
├── train/
│   ├── images/
│   └── labels/
└── valid/
    ├── images/
    └── labels/

data.yaml:

train: C:/Users/boki/repos/poe2trade/training-data/boss-dataset/train/images
val: C:/Users/boki/repos/poe2trade/training-data/boss-dataset/valid/images

nc: 2
names: ['kulemak', 'arbiter']

Copy images and their .txt label files into the appropriate split directories.

4. Train

python train.py --data C:/Users/boki/repos/poe2trade/training-data/boss-dataset/data.yaml --name boss-v1 --epochs 100

Options: --batch 16, --imgsz 640, --device 0 (GPU) or --device cpu.

Best weights are auto-copied to models/boss-v1.pt. If ultralytics auto-increments the run name (e.g. boss-v12), copy manually:

cp runs/detect/boss-v1*/weights/best.pt models/boss-v1.pt

5. Live Detection

The C# app loads models/boss-v1.pt via daemon.py (JSON stdin/stdout protocol). Enable detection in the Mapping tab. Boss bounding boxes render on the Direct2D overlay in cyan.

Files

File	Purpose
annotate.py	Interactive annotator with predict, select/move/resize, tag, filter
train.py	Train YOLOv11n, copies best weights to models/
daemon.py	Persistent inference daemon (managed by C# PythonDetectBridge)
fix_labels.py	One-time fix for labels generated with the old buggy annotator
models/	Trained .pt weight files