refactor
This commit is contained in:
parent
2d6a6bd3a1
commit
d80e723b94
28 changed files with 1801 additions and 352 deletions
197
ARCHITECTURE.md
Normal file
197
ARCHITECTURE.md
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
# POE2 Trade Bot — Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
Automated Path of Exile 2 trade bot with real-time minimap navigation. Monitors the trade site via WebSocket, travels to sellers, buys items from public stash tabs, and stores them. Built with .NET 8.0, Avalonia GUI, Playwright browser automation, and OpenCV screen vision.
|
||||
|
||||
**Target:** `net8.0-windows10.0.19041.0` · **Resolution:** 2560×1440
|
||||
|
||||
---
|
||||
|
||||
## Project Dependency Graph
|
||||
|
||||
```
|
||||
Poe2Trade.Ui (Avalonia WinExe)
|
||||
│
|
||||
Poe2Trade.Bot
|
||||
╱ │ │ ╲
|
||||
Navigation Inventory Trade Log
|
||||
│ │ │ │ │
|
||||
Screen Game │ (Playwright)
|
||||
│ │ │
|
||||
└────┴───────┘
|
||||
│
|
||||
Core
|
||||
```
|
||||
|
||||
| Project | Purpose |
|
||||
|---------|---------|
|
||||
| **Core** | Shared types, enums, config persistence, logging setup |
|
||||
| **Game** | Win32 P/Invoke — window focus, SendInput, clipboard |
|
||||
| **Screen** | DXGI/GDI capture, OpenCV processing, OCR bridge, grid scanning |
|
||||
| **Log** | Polls `Client.txt` for area transitions, whispers, trade events |
|
||||
| **Trade** | Playwright browser automation, WebSocket live search |
|
||||
| **Items** | Ctrl+C clipboard item parsing |
|
||||
| **Inventory** | Grid tracking (5×12), stash deposit, salvage routing |
|
||||
| **Navigation** | Minimap capture, wall detection, world map stitching, BFS exploration |
|
||||
| **Bot** | Orchestration hub, trade executor state machine, trade queue |
|
||||
| **Ui** | Avalonia 11.2 desktop app, MVVM via CommunityToolkit, DI container |
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Trading
|
||||
|
||||
```
|
||||
Trade Site WebSocket ("new" items)
|
||||
↓ NewListings event
|
||||
BotOrchestrator
|
||||
↓ enqueue
|
||||
TradeQueue (FIFO, deduplicates item IDs)
|
||||
↓ dequeue
|
||||
TradeExecutor state machine
|
||||
├─ ClickTravelToHideout (Playwright)
|
||||
├─ FocusGame + wait for area transition (Client.txt)
|
||||
├─ ScanStash (grid template matching)
|
||||
├─ Ctrl+Right-click items (SendInput)
|
||||
├─ /hideout → wait for area transition
|
||||
└─ DepositItemsToStash (inventory grid scan)
|
||||
```
|
||||
|
||||
**States:** Idle → Traveling → InSellersHideout → ScanningStash → Buying → WaitingForMore → GoingHome → InHideout
|
||||
|
||||
### Map Exploration
|
||||
|
||||
```
|
||||
NavigationExecutor.RunExploreLoop (capture loop)
|
||||
├─ FramePipeline.ProcessOneFrame (DXGI / GDI)
|
||||
├─ MinimapCapture.Process (HSV → wall mask, fog mask, player centroid)
|
||||
├─ WorldMap.MatchAndStitch (template match → blit onto 4000×4000 canvas)
|
||||
├─ WorldMap.FindNearestUnexplored (BFS → frontier counting)
|
||||
└─ Post direction to input loop (volatile fields)
|
||||
|
||||
RunInputLoop (concurrent)
|
||||
├─ WASD key holds from direction vector
|
||||
└─ Periodic combat clicks (left + right)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Interfaces
|
||||
|
||||
| Interface | Impl | Role |
|
||||
|-----------|------|------|
|
||||
| `IGameController` | `GameController` | Window focus, mouse/keyboard via P/Invoke SendInput |
|
||||
| `IScreenReader` | `ScreenReader` | Capture, OCR, template match, grid scan |
|
||||
| `IScreenCapture` | `DesktopDuplication` / `GdiCapture` | Raw screen frame acquisition |
|
||||
| `IFrameConsumer` | `MinimapCapture` | Processes frames from FramePipeline |
|
||||
| `ITradeMonitor` | `TradeMonitor` | Playwright persistent context, WebSocket events |
|
||||
| `IClientLogWatcher` | `ClientLogWatcher` | Polls Client.txt (200ms), fires area/whisper/trade events |
|
||||
| `IInventoryManager` | `InventoryManager` | 12×5 grid tracking, stash deposit, salvage |
|
||||
|
||||
---
|
||||
|
||||
## Screen Capture Pipeline
|
||||
|
||||
```
|
||||
IScreenCapture (DXGI preferred, GDI fallback)
|
||||
↓ ScreenFrame (Mat BGRA)
|
||||
FramePipeline.ProcessOneFrame()
|
||||
↓ dispatches to IFrameConsumer(s)
|
||||
MinimapCapture.Process()
|
||||
├─ Mode detection (overlay vs corner minimap)
|
||||
├─ HSV thresholding → player mask, wall mask, fog mask
|
||||
├─ Connected component filtering (min area)
|
||||
├─ Wall color adaptation (per-map tint tracking)
|
||||
└─ MinimapFrame (classified mat, wall mask, gray for correlation)
|
||||
```
|
||||
|
||||
**DXGI path:** AcquireNextFrame → CopySubresourceRegion to staging → Map → memcpy to CPU Mat → Unmap + ReleaseFrame immediately (< 1ms hold).
|
||||
|
||||
---
|
||||
|
||||
## OCR & Grid Detection
|
||||
|
||||
**OCR Engine:** Python EasyOCR daemon (stdin/stdout JSON IPC via `PythonOcrBridge`).
|
||||
|
||||
**Tooltip Detection:** Snapshot-diff approach — capture reference frame, hover item, diff to find darkened overlay rectangle via row/column density analysis.
|
||||
|
||||
**Grid Scanning:** Template match cells against `empty35.png` / `empty70.png` (MAD threshold=2). Item size detection via border comparison + union-find. Visual matching (NCC ≥ 0.70) to identify identical items.
|
||||
|
||||
| Grid | Region | Cell Size |
|
||||
|------|--------|-----------|
|
||||
| Inventory (12×5) | 1696, 788, 840×350 | 70×70 |
|
||||
| Stash (12×12) | 23, 169, 840×840 | 70×70 |
|
||||
| Seller stash | 416, 299, 840×840 | 70×70 |
|
||||
| Stash (24×24) | 23, 169, 840×840 | 35×35 |
|
||||
|
||||
---
|
||||
|
||||
## World Map & Navigation
|
||||
|
||||
`WorldMap` maintains a 4000×4000 byte canvas (`MapCell` enum: Unknown, Explored, Wall, Fog).
|
||||
|
||||
**Localization:** Template matching (NCC) of current minimap frame against the canvas, plus player-offset correction. Falls back to gray-frame correlation tracking.
|
||||
|
||||
**Exploration:** Full BFS from current position. Propagates first-step direction to every reachable cell. Counts frontier cells (Unknown neighbors of Explored) per first-step direction. Picks the direction with the most frontier cells — prefers corridors over dead ends.
|
||||
|
||||
**Stuck Recovery:** If position unchanged for N frames, re-plan with shorter intervals.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
`ConfigStore` persists to `config.json` (JSON with camelCase enums).
|
||||
|
||||
```
|
||||
SavedSettings
|
||||
├── Paused, Headless, Mode (Trading / Mapping)
|
||||
├── Links[] (Url, Name, Active, Mode, PostAction)
|
||||
├── Poe2LogPath, Poe2WindowTitle, BrowserUserDataDir
|
||||
├── TravelTimeoutMs, StashScanTimeoutMs, WaitForMoreItemsMs
|
||||
├── BetweenTradesDelayMs
|
||||
└── WindowX/Y/Width/Height (persisted window geometry)
|
||||
```
|
||||
|
||||
`LinkManager` manages in-memory trade links with config sync. Extracts search ID and league label from URLs.
|
||||
|
||||
---
|
||||
|
||||
## UI (Avalonia)
|
||||
|
||||
- **Theme:** Fluent dark
|
||||
- **Pattern:** MVVM via `CommunityToolkit.Mvvm` (`[ObservableProperty]`, `[RelayCommand]`)
|
||||
- **DI:** `Microsoft.Extensions.DependencyInjection` in `App.axaml.cs`
|
||||
- **Views:** MainWindow (status + logs + links), Settings, Debug (pipeline stages), Mapping (live minimap)
|
||||
- **Log display:** Last 500 entries, color-coded by level
|
||||
|
||||
---
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| **KEYEVENTF_SCANCODE** | Games read hardware scan codes, not virtual key codes |
|
||||
| **Persistent Playwright context** | Preserves trade site login across restarts |
|
||||
| **Client.txt polling** | More reliable than OCR for detecting area transitions |
|
||||
| **DXGI Desktop Duplication** | ~4ms full-screen capture vs ~15ms GDI; early frame release prevents DWM stalls |
|
||||
| **Separate capture + input loops** | Capture decisions never blocked by input latency (mouse moves, key holds) |
|
||||
| **Volatile fields for loop communication** | Lock-free direction posting between capture and input loops |
|
||||
| **BFS frontier counting** | Avoids dead-end rooms by preferring directions with more unexplored cells |
|
||||
| **Wall color adaptation** | Each map has slightly different tint; tracks per-frame HSV statistics |
|
||||
| **Snapshot-diff OCR** | Detects tooltips/dialogs by comparing against clean reference frame |
|
||||
|
||||
---
|
||||
|
||||
## External Dependencies
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| Avalonia 11.2 | Desktop UI framework |
|
||||
| CommunityToolkit.Mvvm | MVVM source generators |
|
||||
| Microsoft.Playwright | Browser automation + WebSocket |
|
||||
| OpenCvSharp4 | Image processing, template matching, NCC |
|
||||
| Vortice.Direct3D11 / DXGI | DXGI Desktop Duplication capture |
|
||||
| System.Drawing.Common | GDI fallback capture |
|
||||
| Serilog | Structured logging (console + rolling file) |
|
||||
|
|
@ -24,8 +24,8 @@ public class BotOrchestrator : IAsyncDisposable
|
|||
{
|
||||
private bool _paused;
|
||||
private string _state = "Idle";
|
||||
private int _tradesCompleted;
|
||||
private int _tradesFailed;
|
||||
private volatile int _tradesCompleted;
|
||||
private volatile int _tradesFailed;
|
||||
private readonly long _startTime = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
|
||||
private bool _started;
|
||||
|
||||
|
|
@ -40,6 +40,11 @@ public class BotOrchestrator : IAsyncDisposable
|
|||
public TradeExecutor TradeExecutor { get; }
|
||||
public TradeQueue TradeQueue { get; }
|
||||
public NavigationExecutor Navigation { get; }
|
||||
public FramePipelineService PipelineService { get; }
|
||||
public GameStateDetector GameState { get; }
|
||||
public HudReader HudReader { get; }
|
||||
public EnemyDetector EnemyDetector { get; }
|
||||
public FrameSaver FrameSaver { get; }
|
||||
private readonly Dictionary<string, ScrapExecutor> _scrapExecutors = new();
|
||||
|
||||
// Events
|
||||
|
|
@ -49,7 +54,7 @@ public class BotOrchestrator : IAsyncDisposable
|
|||
public BotOrchestrator(ConfigStore store, IGameController game, IScreenReader screen,
|
||||
IClientLogWatcher logWatcher, ITradeMonitor tradeMonitor,
|
||||
IInventoryManager inventory, TradeExecutor tradeExecutor,
|
||||
TradeQueue tradeQueue, LinkManager links)
|
||||
TradeQueue tradeQueue, LinkManager links, FramePipelineService pipelineService)
|
||||
{
|
||||
Store = store;
|
||||
Game = game;
|
||||
|
|
@ -60,7 +65,26 @@ public class BotOrchestrator : IAsyncDisposable
|
|||
TradeExecutor = tradeExecutor;
|
||||
TradeQueue = tradeQueue;
|
||||
Links = links;
|
||||
Navigation = new NavigationExecutor(game);
|
||||
PipelineService = pipelineService;
|
||||
|
||||
// Create consumers
|
||||
var minimapCapture = new MinimapCapture(new MinimapConfig(), pipelineService.Backend);
|
||||
GameState = new GameStateDetector();
|
||||
HudReader = new HudReader();
|
||||
EnemyDetector = new EnemyDetector();
|
||||
FrameSaver = new FrameSaver();
|
||||
|
||||
// Register on shared pipeline
|
||||
pipelineService.Pipeline.AddConsumer(minimapCapture);
|
||||
pipelineService.Pipeline.AddConsumer(GameState);
|
||||
pipelineService.Pipeline.AddConsumer(HudReader);
|
||||
pipelineService.Pipeline.AddConsumer(EnemyDetector);
|
||||
pipelineService.Pipeline.AddConsumer(FrameSaver);
|
||||
|
||||
// Pass shared pipeline to NavigationExecutor
|
||||
Navigation = new NavigationExecutor(game, pipelineService.Pipeline, minimapCapture,
|
||||
enemyDetector: EnemyDetector);
|
||||
|
||||
logWatcher.AreaEntered += _ => Navigation.Reset();
|
||||
logWatcher.Start(); // start early so area events fire even before Bot.Start()
|
||||
_paused = store.Settings.Paused;
|
||||
|
|
@ -205,8 +229,8 @@ public class BotOrchestrator : IAsyncDisposable
|
|||
// Wire executor events
|
||||
TradeExecutor.StateChanged += _ => UpdateExecutorState();
|
||||
Navigation.StateChanged += _ => UpdateExecutorState();
|
||||
TradeQueue.TradeCompleted += () => { _tradesCompleted++; StatusUpdated?.Invoke(); };
|
||||
TradeQueue.TradeFailed += () => { _tradesFailed++; StatusUpdated?.Invoke(); };
|
||||
TradeQueue.TradeCompleted += () => { Interlocked.Increment(ref _tradesCompleted); StatusUpdated?.Invoke(); };
|
||||
TradeQueue.TradeFailed += () => { Interlocked.Increment(ref _tradesFailed); StatusUpdated?.Invoke(); };
|
||||
Inventory.Updated += () => StatusUpdated?.Invoke();
|
||||
|
||||
_started = true;
|
||||
|
|
@ -267,9 +291,11 @@ public class BotOrchestrator : IAsyncDisposable
|
|||
Log.Information("Shutting down bot...");
|
||||
foreach (var exec in _scrapExecutors.Values)
|
||||
await exec.Stop();
|
||||
EnemyDetector.Dispose();
|
||||
Screen.Dispose();
|
||||
await TradeMonitor.DisposeAsync();
|
||||
LogWatcher.Dispose();
|
||||
PipelineService.Dispose();
|
||||
}
|
||||
|
||||
private void OnNewListings(string searchId, List<string> itemIds, IPage page)
|
||||
|
|
@ -302,8 +328,8 @@ public class BotOrchestrator : IAsyncDisposable
|
|||
{
|
||||
var scrapExec = new ScrapExecutor(Game, Screen, TradeMonitor, Inventory, Config);
|
||||
scrapExec.StateChanged += _ => UpdateExecutorState();
|
||||
scrapExec.ItemBought += () => { _tradesCompleted++; StatusUpdated?.Invoke(); };
|
||||
scrapExec.ItemFailed += () => { _tradesFailed++; StatusUpdated?.Invoke(); };
|
||||
scrapExec.ItemBought += () => { Interlocked.Increment(ref _tradesCompleted); StatusUpdated?.Invoke(); };
|
||||
scrapExec.ItemFailed += () => { Interlocked.Increment(ref _tradesFailed); StatusUpdated?.Invoke(); };
|
||||
_scrapExecutors[link.Id] = scrapExec;
|
||||
Emit("info", $"Scrap loop started: {link.Name}");
|
||||
StatusUpdated?.Invoke();
|
||||
|
|
|
|||
|
|
@ -81,3 +81,22 @@ public enum MapType
|
|||
Temple,
|
||||
Endgame
|
||||
}
|
||||
|
||||
public enum GameUiState
|
||||
{
|
||||
Unknown,
|
||||
Playing,
|
||||
InventoryOpen,
|
||||
StashOpen,
|
||||
VendorOpen,
|
||||
MapOverlay,
|
||||
Loading,
|
||||
Dead,
|
||||
EscapeMenu
|
||||
}
|
||||
|
||||
public interface IGameStateProvider
|
||||
{
|
||||
GameUiState CurrentState { get; }
|
||||
event Action<GameUiState, GameUiState>? StateChanged;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,11 +8,15 @@ namespace Poe2Trade.Game;
|
|||
/// </summary>
|
||||
public static class ClipboardHelper
|
||||
{
|
||||
private const int MaxRetries = 5;
|
||||
private const int RetryDelayMs = 30;
|
||||
|
||||
public static string Read()
|
||||
{
|
||||
if (!ClipboardNative.OpenClipboard(IntPtr.Zero))
|
||||
return "";
|
||||
|
||||
for (var attempt = 0; attempt < MaxRetries; attempt++)
|
||||
{
|
||||
if (ClipboardNative.OpenClipboard(IntPtr.Zero))
|
||||
{
|
||||
try
|
||||
{
|
||||
var handle = ClipboardNative.GetClipboardData(ClipboardNative.CF_UNICODETEXT);
|
||||
|
|
@ -35,12 +39,17 @@ public static class ClipboardHelper
|
|||
ClipboardNative.CloseClipboard();
|
||||
}
|
||||
}
|
||||
Thread.Sleep(RetryDelayMs);
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
public static void Write(string text)
|
||||
{
|
||||
if (!ClipboardNative.OpenClipboard(IntPtr.Zero))
|
||||
return;
|
||||
|
||||
for (var attempt = 0; attempt < MaxRetries; attempt++)
|
||||
{
|
||||
if (ClipboardNative.OpenClipboard(IntPtr.Zero))
|
||||
{
|
||||
try
|
||||
{
|
||||
ClipboardNative.EmptyClipboard();
|
||||
|
|
@ -65,12 +74,16 @@ public static class ClipboardHelper
|
|||
}
|
||||
|
||||
ClipboardNative.SetClipboardData(ClipboardNative.CF_UNICODETEXT, hGlobal);
|
||||
return;
|
||||
}
|
||||
finally
|
||||
{
|
||||
ClipboardNative.CloseClipboard();
|
||||
}
|
||||
}
|
||||
Thread.Sleep(RetryDelayMs);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
internal static partial class ClipboardNative
|
||||
|
|
|
|||
|
|
@ -13,59 +13,67 @@ public class NavigationExecutor : IDisposable
|
|||
private readonly FramePipeline _pipeline;
|
||||
private readonly MinimapCapture _capture;
|
||||
private readonly WorldMap _worldMap;
|
||||
private readonly StuckDetector _stuck;
|
||||
private readonly EnemyDetector? _enemyDetector;
|
||||
private NavigationState _state = NavigationState.Idle;
|
||||
private volatile bool _stopped;
|
||||
private int _stuckCounter;
|
||||
private MapPosition? _lastPosition;
|
||||
private volatile byte[]? _cachedViewport;
|
||||
private static readonly Random Rng = new();
|
||||
|
||||
// Input loop communication (capture loop writes, input loop reads)
|
||||
// Thread-safe direction passing (capture loop writes, input loop reads)
|
||||
private readonly object _dirLock = new();
|
||||
private double _desiredDirX, _desiredDirY;
|
||||
private volatile bool _directionChanged;
|
||||
private bool _directionChanged;
|
||||
|
||||
// Valid state transitions (from → allowed targets)
|
||||
private static readonly Dictionary<NavigationState, NavigationState[]> ValidTransitions = new()
|
||||
{
|
||||
[NavigationState.Idle] = [NavigationState.Capturing],
|
||||
[NavigationState.Capturing] = [NavigationState.Processing, NavigationState.Idle],
|
||||
[NavigationState.Processing] = [NavigationState.Planning, NavigationState.Stuck, NavigationState.Capturing, NavigationState.Failed, NavigationState.Idle],
|
||||
[NavigationState.Planning] = [NavigationState.Moving, NavigationState.Completed, NavigationState.Idle],
|
||||
[NavigationState.Moving] = [NavigationState.Capturing, NavigationState.Idle],
|
||||
[NavigationState.Stuck] = [NavigationState.Moving, NavigationState.Completed, NavigationState.Idle],
|
||||
[NavigationState.Completed] = [NavigationState.Idle],
|
||||
[NavigationState.Failed] = [NavigationState.Capturing, NavigationState.Idle],
|
||||
};
|
||||
|
||||
public event Action<NavigationState>? StateChanged;
|
||||
public event Action<byte[]>? ViewportUpdated;
|
||||
public NavigationState State => _state;
|
||||
|
||||
public NavigationExecutor(IGameController game, MinimapConfig? config = null)
|
||||
public NavigationExecutor(IGameController game, FramePipeline pipeline,
|
||||
MinimapCapture capture, MinimapConfig? config = null, EnemyDetector? enemyDetector = null)
|
||||
{
|
||||
_game = game;
|
||||
_config = config ?? new MinimapConfig();
|
||||
|
||||
var backend = CreateBackend();
|
||||
_pipeline = new FramePipeline(backend);
|
||||
_capture = new MinimapCapture(_config, backend);
|
||||
_pipeline.AddConsumer(_capture);
|
||||
_stuck = new StuckDetector(_config.StuckThreshold, _config.StuckFrameCount);
|
||||
_pipeline = pipeline;
|
||||
_capture = capture;
|
||||
_enemyDetector = enemyDetector;
|
||||
|
||||
_worldMap = new WorldMap(_config);
|
||||
_capture.ModeChanged += _ =>
|
||||
{
|
||||
_worldMap.Rebootstrap();
|
||||
_stuckCounter = 0;
|
||||
_lastPosition = null;
|
||||
_stuck.Reset();
|
||||
};
|
||||
}
|
||||
|
||||
private static IScreenCapture CreateBackend()
|
||||
{
|
||||
// DXGI primary → GDI fallback
|
||||
try
|
||||
{
|
||||
var dxgi = new DesktopDuplication();
|
||||
Log.Information("Screen capture: DXGI Desktop Duplication");
|
||||
return dxgi;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Log.Warning(ex, "DXGI unavailable, falling back to GDI");
|
||||
}
|
||||
|
||||
Log.Information("Screen capture: GDI (CopyFromScreen)");
|
||||
return new GdiCapture();
|
||||
}
|
||||
|
||||
private void SetState(NavigationState s)
|
||||
{
|
||||
var from = _state;
|
||||
if (from == s) return;
|
||||
|
||||
// Validate transition (Stop() can set Idle from any state — always allowed)
|
||||
if (s != NavigationState.Idle &&
|
||||
ValidTransitions.TryGetValue(from, out var allowed) &&
|
||||
!allowed.Contains(s))
|
||||
{
|
||||
Log.Warning("Invalid state transition: {From} → {To}", from, s);
|
||||
}
|
||||
|
||||
Log.Debug("State: {From} → {To}", from, s);
|
||||
_state = s;
|
||||
StateChanged?.Invoke(s);
|
||||
}
|
||||
|
|
@ -83,8 +91,7 @@ public class NavigationExecutor : IDisposable
|
|||
_worldMap.Reset();
|
||||
_capture.ResetAdaptation();
|
||||
_stopped = false;
|
||||
_stuckCounter = 0;
|
||||
_lastPosition = null;
|
||||
_stuck.Reset();
|
||||
SetState(NavigationState.Idle);
|
||||
Log.Information("Navigation reset (new area)");
|
||||
}
|
||||
|
|
@ -92,7 +99,7 @@ public class NavigationExecutor : IDisposable
|
|||
public async Task RunExploreLoop()
|
||||
{
|
||||
_stopped = false;
|
||||
_directionChanged = false;
|
||||
lock (_dirLock) _directionChanged = false;
|
||||
Log.Information("Starting explore loop");
|
||||
_cachedViewport = _worldMap.GetViewportSnapshot(_worldMap.Position);
|
||||
|
||||
|
|
@ -127,31 +134,21 @@ public class NavigationExecutor : IDisposable
|
|||
_capture.CommitWallColors();
|
||||
// Only re-render viewport when canvas was modified (avoids ~3ms PNG encode on dedup-skips)
|
||||
_cachedViewport = _worldMap.GetViewportSnapshot(pos);
|
||||
ViewportUpdated?.Invoke(_cachedViewport);
|
||||
}
|
||||
|
||||
// Stuck detection
|
||||
if (_lastPosition != null)
|
||||
{
|
||||
var dx = pos.X - _lastPosition.X;
|
||||
var dy = pos.Y - _lastPosition.Y;
|
||||
if (Math.Sqrt(dx * dx + dy * dy) < _config.StuckThreshold)
|
||||
_stuckCounter++;
|
||||
else
|
||||
_stuckCounter = 0;
|
||||
}
|
||||
_lastPosition = pos;
|
||||
_stuck.Update(pos);
|
||||
|
||||
// 2. Movement decisions — non-blocking, just post direction to input loop
|
||||
var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
|
||||
var moveInterval = _stuckCounter >= _config.StuckFrameCount
|
||||
? 200
|
||||
: _config.MovementWaitMs;
|
||||
var moveInterval = _stuck.IsStuck ? 200 : _config.MovementWaitMs;
|
||||
|
||||
if (now - lastMoveTime >= moveInterval)
|
||||
{
|
||||
lastMoveTime = now;
|
||||
|
||||
if (_stuckCounter >= _config.StuckFrameCount)
|
||||
if (_stuck.IsStuck)
|
||||
SetState(NavigationState.Stuck);
|
||||
else
|
||||
SetState(NavigationState.Planning);
|
||||
|
|
@ -167,13 +164,16 @@ public class NavigationExecutor : IDisposable
|
|||
}
|
||||
|
||||
SetState(NavigationState.Moving);
|
||||
// Post direction to input loop (non-blocking)
|
||||
// Post direction to input loop (thread-safe)
|
||||
lock (_dirLock)
|
||||
{
|
||||
_desiredDirX = direction.Value.dirX;
|
||||
_desiredDirY = direction.Value.dirY;
|
||||
_directionChanged = true;
|
||||
}
|
||||
|
||||
if (_stuckCounter >= _config.StuckFrameCount)
|
||||
_stuckCounter = 0; // reset after re-routing
|
||||
if (_stuck.IsStuck)
|
||||
_stuck.Reset(); // reset after re-routing
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
|
|
@ -217,28 +217,44 @@ public class NavigationExecutor : IDisposable
|
|||
{
|
||||
while (!_stopped)
|
||||
{
|
||||
// Apply direction changes from capture loop
|
||||
if (_directionChanged)
|
||||
// Apply direction changes from capture loop (thread-safe read)
|
||||
bool changed;
|
||||
double dirX, dirY;
|
||||
lock (_dirLock)
|
||||
{
|
||||
changed = _directionChanged;
|
||||
dirX = _desiredDirX;
|
||||
dirY = _desiredDirY;
|
||||
_directionChanged = false;
|
||||
var dirX = _desiredDirX;
|
||||
var dirY = _desiredDirY;
|
||||
await UpdateWasdKeys(heldKeys, dirX, dirY);
|
||||
}
|
||||
|
||||
// Combat clicks on timer
|
||||
if (changed)
|
||||
await UpdateWasdKeys(heldKeys, dirX, dirY);
|
||||
|
||||
// Combat clicks on timer — prefer detected enemies, fall back to random
|
||||
var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
|
||||
if (now >= nextCombatTime)
|
||||
{
|
||||
nextCombatTime = now + 1000 + Rng.Next(1000);
|
||||
var cx = 1280 + Rng.Next(-150, 150);
|
||||
var cy = 720 + Rng.Next(-150, 150);
|
||||
|
||||
var target = GetBestTarget();
|
||||
if (target != null)
|
||||
{
|
||||
await _game.LeftClickAt(target.Cx, target.Cy);
|
||||
await Helpers.Sleep(100 + Rng.Next(100));
|
||||
await _game.RightClickAt(target.Cx, target.Cy);
|
||||
}
|
||||
else
|
||||
{
|
||||
var cx = _config.ScreenCenterX + Rng.Next(-150, 150);
|
||||
var cy = _config.ScreenCenterY + Rng.Next(-150, 150);
|
||||
await _game.LeftClickAt(cx, cy);
|
||||
await Helpers.Sleep(100 + Rng.Next(100));
|
||||
cx = 1280 + Rng.Next(-150, 150);
|
||||
cy = 720 + Rng.Next(-150, 150);
|
||||
cx = _config.ScreenCenterX + Rng.Next(-150, 150);
|
||||
cy = _config.ScreenCenterY + Rng.Next(-150, 150);
|
||||
await _game.RightClickAt(cx, cy);
|
||||
}
|
||||
}
|
||||
|
||||
await Helpers.Sleep(15);
|
||||
}
|
||||
|
|
@ -288,6 +304,26 @@ public class NavigationExecutor : IDisposable
|
|||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pick the best enemy target from detection snapshot.
|
||||
/// Prefers health-bar-confirmed enemies, then closest to screen center.
|
||||
/// </summary>
|
||||
private DetectedEnemy? GetBestTarget()
|
||||
{
|
||||
if (_enemyDetector == null) return null;
|
||||
|
||||
var snapshot = _enemyDetector.Latest;
|
||||
if (snapshot.Enemies.Count == 0) return null;
|
||||
|
||||
var screenCx = _config.ScreenCenterX;
|
||||
var screenCy = _config.ScreenCenterY;
|
||||
|
||||
return snapshot.Enemies
|
||||
.OrderByDescending(e => e.HealthBarConfirmed)
|
||||
.ThenBy(e => Math.Abs(e.Cx - screenCx) + Math.Abs(e.Cy - screenCy))
|
||||
.FirstOrDefault();
|
||||
}
|
||||
|
||||
private async Task ClickToMove(double dirX, double dirY)
|
||||
{
|
||||
// Player is at minimap center on screen; click offset from center
|
||||
|
|
@ -373,7 +409,6 @@ public class NavigationExecutor : IDisposable
|
|||
public void Dispose()
|
||||
{
|
||||
_capture.Dispose();
|
||||
_pipeline.Dispose();
|
||||
_worldMap.Dispose();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -126,7 +126,7 @@ public class MinimapConfig
|
|||
public int MovementWaitMs { get; set; } = 600;
|
||||
|
||||
// World map canvas
|
||||
public int CanvasSize { get; set; } = 4000;
|
||||
public int CanvasSize { get; set; } = 2000;
|
||||
|
||||
// Template matching: search radius around current position estimate (pixels)
|
||||
public int MatchSearchRadius { get; set; } = 100;
|
||||
|
|
@ -153,4 +153,8 @@ public class MinimapConfig
|
|||
// Stuck detection
|
||||
public double StuckThreshold { get; set; } = 2.0;
|
||||
public int StuckFrameCount { get; set; } = 5;
|
||||
|
||||
// Screen center for combat clicks (2560x1440 → center = 1280, 720)
|
||||
public int ScreenCenterX { get; set; } = 1280;
|
||||
public int ScreenCenterY { get; set; } = 720;
|
||||
}
|
||||
|
|
|
|||
145
src/Poe2Trade.Navigation/PathFinder.cs
Normal file
145
src/Poe2Trade.Navigation/PathFinder.cs
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
using OpenCvSharp;
|
||||
using Serilog;
|
||||
|
||||
namespace Poe2Trade.Navigation;
|
||||
|
||||
/// <summary>
|
||||
/// BFS pathfinding through the world map canvas. Pure function — reads canvas, never modifies it.
|
||||
/// </summary>
|
||||
internal class PathFinder
|
||||
{
|
||||
/// <summary>
|
||||
/// BFS through walkable (Explored) cells to find the best frontier direction.
|
||||
/// Instead of stopping at the nearest frontier, runs the full BFS and counts
|
||||
/// frontier cells reachable per first-step direction. Prefers directions with
|
||||
/// more frontier cells (corridors) over directions with few (dead-end rooms).
|
||||
/// </summary>
|
||||
public (double dirX, double dirY)? FindNearestUnexplored(Mat canvas, int canvasSize, MapPosition pos, int searchRadius = 400)
|
||||
{
|
||||
var cx = (int)Math.Round(pos.X);
|
||||
var cy = (int)Math.Round(pos.Y);
|
||||
|
||||
// BFS at half resolution for speed (step=2 → ~200x200 effective grid for r=400)
|
||||
const int step = 2;
|
||||
var size = canvasSize;
|
||||
var rr = searchRadius / step;
|
||||
var gridW = 2 * rr + 1;
|
||||
|
||||
var visited = new bool[gridW * gridW];
|
||||
// Propagate first step from start during BFS (avoids per-frontier trace-back)
|
||||
var firstStepX = new short[gridW * gridW];
|
||||
var firstStepY = new short[gridW * gridW];
|
||||
|
||||
var queue = new Queue<(int gx, int gy)>(4096);
|
||||
var startGx = rr;
|
||||
var startGy = rr;
|
||||
visited[startGy * gridW + startGx] = true;
|
||||
queue.Enqueue((startGx, startGy));
|
||||
|
||||
// 8-connected neighbors
|
||||
ReadOnlySpan<int> dxs = [-1, 0, 1, -1, 1, -1, 0, 1];
|
||||
ReadOnlySpan<int> dys = [-1, -1, -1, 0, 0, 1, 1, 1];
|
||||
|
||||
// Count frontier cells per first-step direction
|
||||
var frontierCounts = new Dictionary<int, int>();
|
||||
var firstStepCoords = new Dictionary<int, (short gx, short gy)>();
|
||||
|
||||
while (queue.Count > 0)
|
||||
{
|
||||
var (gx, gy) = queue.Dequeue();
|
||||
|
||||
// Map grid coords back to canvas coords
|
||||
var wx = cx + (gx - rr) * step;
|
||||
var wy = cy + (gy - rr) * step;
|
||||
|
||||
// Check if this explored cell borders Unknown/Fog (= frontier)
|
||||
if (gx != startGx || gy != startGy)
|
||||
{
|
||||
var cellIdx = gy * gridW + gx;
|
||||
for (var d = 0; d < 8; d++)
|
||||
{
|
||||
var nx = wx + dxs[d] * step;
|
||||
var ny = wy + dys[d] * step;
|
||||
if (nx < 0 || nx >= size || ny < 0 || ny >= size) continue;
|
||||
var neighbor = canvas.At<byte>(ny, nx);
|
||||
if (neighbor == (byte)MapCell.Unknown || neighbor == (byte)MapCell.Fog)
|
||||
{
|
||||
var fsKey = firstStepY[cellIdx] * gridW + firstStepX[cellIdx];
|
||||
if (frontierCounts.TryGetValue(fsKey, out var cnt))
|
||||
frontierCounts[fsKey] = cnt + 1;
|
||||
else
|
||||
{
|
||||
frontierCounts[fsKey] = 1;
|
||||
firstStepCoords[fsKey] = (firstStepX[cellIdx], firstStepY[cellIdx]);
|
||||
}
|
||||
break; // don't double-count this cell
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Expand to walkable neighbors
|
||||
for (var d = 0; d < 8; d++)
|
||||
{
|
||||
var ngx = gx + dxs[d];
|
||||
var ngy = gy + dys[d];
|
||||
if (ngx < 0 || ngx >= gridW || ngy < 0 || ngy >= gridW) continue;
|
||||
|
||||
var idx = ngy * gridW + ngx;
|
||||
if (visited[idx]) continue;
|
||||
|
||||
var nwx = cx + (ngx - rr) * step;
|
||||
var nwy = cy + (ngy - rr) * step;
|
||||
if (nwx < 0 || nwx >= size || nwy < 0 || nwy >= size) continue;
|
||||
|
||||
var cell = canvas.At<byte>(nwy, nwx);
|
||||
if (cell != (byte)MapCell.Explored) continue;
|
||||
|
||||
visited[idx] = true;
|
||||
// Propagate first step: direct neighbors of start ARE the first step
|
||||
if (gx == startGx && gy == startGy)
|
||||
{
|
||||
firstStepX[idx] = (short)ngx;
|
||||
firstStepY[idx] = (short)ngy;
|
||||
}
|
||||
else
|
||||
{
|
||||
var parentIdx = gy * gridW + gx;
|
||||
firstStepX[idx] = firstStepX[parentIdx];
|
||||
firstStepY[idx] = firstStepY[parentIdx];
|
||||
}
|
||||
queue.Enqueue((ngx, ngy));
|
||||
}
|
||||
}
|
||||
|
||||
if (frontierCounts.Count == 0)
|
||||
{
|
||||
Log.Information("BFS: no reachable frontier within {Radius}px", searchRadius);
|
||||
return null;
|
||||
}
|
||||
|
||||
// Pick direction with the most frontier cells (prefers corridors over dead ends)
|
||||
var bestKey = -1;
|
||||
var bestCount = 0;
|
||||
foreach (var (key, count) in frontierCounts)
|
||||
{
|
||||
if (count > bestCount)
|
||||
{
|
||||
bestCount = count;
|
||||
bestKey = key;
|
||||
}
|
||||
}
|
||||
|
||||
var (bestGx, bestGy) = firstStepCoords[bestKey];
|
||||
var dirX = (double)(bestGx - startGx);
|
||||
var dirY = (double)(bestGy - startGy);
|
||||
var len = Math.Sqrt(dirX * dirX + dirY * dirY);
|
||||
if (len < 0.001) return (1, 0);
|
||||
|
||||
dirX /= len;
|
||||
dirY /= len;
|
||||
|
||||
Log.Debug("BFS: {DirCount} directions, best={Best} frontier cells, dir=({Dx:F2},{Dy:F2})",
|
||||
frontierCounts.Count, bestCount, dirX, dirY);
|
||||
return (dirX, dirY);
|
||||
}
|
||||
}
|
||||
40
src/Poe2Trade.Navigation/StuckDetector.cs
Normal file
40
src/Poe2Trade.Navigation/StuckDetector.cs
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
namespace Poe2Trade.Navigation;
|
||||
|
||||
/// <summary>
|
||||
/// Detects when the player hasn't moved significantly over a window of frames.
|
||||
/// </summary>
|
||||
internal class StuckDetector
|
||||
{
|
||||
private readonly double _threshold;
|
||||
private readonly int _frameCount;
|
||||
private int _counter;
|
||||
private MapPosition? _lastPosition;
|
||||
|
||||
public bool IsStuck => _counter >= _frameCount;
|
||||
|
||||
public StuckDetector(double threshold, int frameCount)
|
||||
{
|
||||
_threshold = threshold;
|
||||
_frameCount = frameCount;
|
||||
}
|
||||
|
||||
public void Update(MapPosition position)
|
||||
{
|
||||
if (_lastPosition != null)
|
||||
{
|
||||
var dx = position.X - _lastPosition.X;
|
||||
var dy = position.Y - _lastPosition.Y;
|
||||
if (Math.Sqrt(dx * dx + dy * dy) < _threshold)
|
||||
_counter++;
|
||||
else
|
||||
_counter = 0;
|
||||
}
|
||||
_lastPosition = position;
|
||||
}
|
||||
|
||||
public void Reset()
|
||||
{
|
||||
_counter = 0;
|
||||
_lastPosition = null;
|
||||
}
|
||||
}
|
||||
|
|
@ -24,6 +24,10 @@ internal class WallColorTracker
|
|||
private readonly int[] _pendV = new int[256];
|
||||
private int _pendCount;
|
||||
|
||||
// Generation counter: prevents committing samples from a previous mode after Reset()
|
||||
private int _generation;
|
||||
private int _pendGeneration;
|
||||
|
||||
private const int MinSamples = 3000;
|
||||
private const double LoPercentile = 0.05;
|
||||
private const double HiPercentile = 0.95;
|
||||
|
|
@ -51,6 +55,7 @@ internal class WallColorTracker
|
|||
Array.Clear(_pendS);
|
||||
Array.Clear(_pendV);
|
||||
_pendCount = 0;
|
||||
_pendGeneration = _generation;
|
||||
|
||||
// Downsample: every 4th pixel in each direction (~10K samples for 400x400)
|
||||
for (var r = 0; r < hsv.Rows; r += 4)
|
||||
|
|
@ -72,6 +77,7 @@ internal class WallColorTracker
|
|||
public void Commit()
|
||||
{
|
||||
if (_pendCount == 0) return;
|
||||
if (_pendGeneration != _generation) return; // stale cross-mode samples
|
||||
|
||||
for (var i = 0; i < 180; i++) _hHist[i] += _pendH[i];
|
||||
for (var i = 0; i < 256; i++) _sHist[i] += _pendS[i];
|
||||
|
|
@ -144,5 +150,6 @@ internal class WallColorTracker
|
|||
_pendCount = 0;
|
||||
AdaptedLo = null;
|
||||
AdaptedHi = null;
|
||||
_generation++;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -7,22 +7,57 @@ namespace Poe2Trade.Navigation;
|
|||
public class WorldMap : IDisposable
|
||||
{
|
||||
private readonly MinimapConfig _config;
|
||||
private readonly Mat _canvas;
|
||||
private readonly Mat _confidence; // CV_16SC1: per-pixel wall confidence counter
|
||||
private Mat _canvas;
|
||||
private Mat _confidence; // CV_16SC1: per-pixel wall confidence counter
|
||||
private int _canvasSize;
|
||||
private MapPosition _position;
|
||||
private int _frameCount;
|
||||
private int _consecutiveMatchFails;
|
||||
private Mat? _prevWallMask; // for frame deduplication
|
||||
private readonly PathFinder _pathFinder = new();
|
||||
|
||||
public MapPosition Position => _position;
|
||||
public bool LastMatchSucceeded { get; private set; }
|
||||
public int CanvasSize => _canvasSize;
|
||||
|
||||
private const int GrowMargin = 500;
|
||||
private const int GrowAmount = 2000; // 1000px added per side
|
||||
|
||||
public WorldMap(MinimapConfig config)
|
||||
{
|
||||
_config = config;
|
||||
_canvas = new Mat(config.CanvasSize, config.CanvasSize, MatType.CV_8UC1, Scalar.Black);
|
||||
_confidence = new Mat(config.CanvasSize, config.CanvasSize, MatType.CV_16SC1, Scalar.Black);
|
||||
_position = new MapPosition(config.CanvasSize / 2.0, config.CanvasSize / 2.0);
|
||||
_canvasSize = config.CanvasSize;
|
||||
_canvas = new Mat(_canvasSize, _canvasSize, MatType.CV_8UC1, Scalar.Black);
|
||||
_confidence = new Mat(_canvasSize, _canvasSize, MatType.CV_16SC1, Scalar.Black);
|
||||
_position = new MapPosition(_canvasSize / 2.0, _canvasSize / 2.0);
|
||||
}
|
||||
|
||||
private void EnsureCapacity()
|
||||
{
|
||||
var x = (int)Math.Round(_position.X);
|
||||
var y = (int)Math.Round(_position.Y);
|
||||
if (x >= GrowMargin && y >= GrowMargin &&
|
||||
x < _canvasSize - GrowMargin && y < _canvasSize - GrowMargin)
|
||||
return;
|
||||
|
||||
var oldSize = _canvasSize;
|
||||
var newSize = oldSize + GrowAmount;
|
||||
var offset = GrowAmount / 2;
|
||||
|
||||
var newCanvas = new Mat(newSize, newSize, MatType.CV_8UC1, Scalar.Black);
|
||||
var newConf = new Mat(newSize, newSize, MatType.CV_16SC1, Scalar.Black);
|
||||
using (var dst = new Mat(newCanvas, new Rect(offset, offset, oldSize, oldSize)))
|
||||
_canvas.CopyTo(dst);
|
||||
using (var dst = new Mat(newConf, new Rect(offset, offset, oldSize, oldSize)))
|
||||
_confidence.CopyTo(dst);
|
||||
|
||||
_canvas.Dispose();
|
||||
_confidence.Dispose();
|
||||
_canvas = newCanvas;
|
||||
_confidence = newConf;
|
||||
_canvasSize = newSize;
|
||||
_position = new MapPosition(_position.X + offset, _position.Y + offset);
|
||||
Log.Information("Canvas grown: {Old}→{New}, offset={Offset}", oldSize, newSize, offset);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
|
@ -31,6 +66,7 @@ public class WorldMap : IDisposable
|
|||
/// </summary>
|
||||
public MapPosition MatchAndStitch(Mat classifiedMat, Mat wallMask, MinimapMode mode = MinimapMode.Overlay)
|
||||
{
|
||||
EnsureCapacity();
|
||||
var sw = Stopwatch.StartNew();
|
||||
_frameCount++;
|
||||
|
||||
|
|
@ -44,35 +80,10 @@ public class WorldMap : IDisposable
|
|||
|
||||
var wallCountBefore = Cv2.CountNonZero(wallMask);
|
||||
|
||||
// Block-based noise filter: only needed for overlay (game effects bleed through)
|
||||
// Skip during warmup — we need walls to seed the canvas, confidence handles noise
|
||||
if (!isCorner && !needsBootstrap)
|
||||
{
|
||||
var cleanFraction = FilterNoisyBlocks(wallMask, classifiedMat);
|
||||
if (cleanFraction < 0.25)
|
||||
{
|
||||
Log.Information("Noise filter: {Clean:P0} clean, skipping ({Ms:F1}ms)",
|
||||
cleanFraction, sw.Elapsed.TotalMilliseconds);
|
||||
if (ShouldSkipFrame(classifiedMat, wallMask, isCorner, needsBootstrap, sw))
|
||||
return _position;
|
||||
}
|
||||
}
|
||||
|
||||
var wallCountAfter = Cv2.CountNonZero(wallMask);
|
||||
|
||||
// Frame deduplication: skip if minimap hasn't scrolled yet (but always allow warmup through)
|
||||
if (!needsBootstrap && _prevWallMask != null && _frameCount > 1)
|
||||
{
|
||||
using var xor = new Mat();
|
||||
Cv2.BitwiseXor(wallMask, _prevWallMask, xor);
|
||||
var changedPixels = Cv2.CountNonZero(xor);
|
||||
if (changedPixels < _config.FrameChangeThreshold)
|
||||
{
|
||||
Log.Debug("Frame dedup: {Changed} changed pixels, skipping ({Ms:F1}ms)",
|
||||
changedPixels, sw.Elapsed.TotalMilliseconds);
|
||||
return _position;
|
||||
}
|
||||
}
|
||||
|
||||
var dedupMs = sw.Elapsed.TotalMilliseconds;
|
||||
|
||||
// Store current wall mask for next frame's dedup check
|
||||
|
|
@ -83,6 +94,7 @@ public class WorldMap : IDisposable
|
|||
if (needsBootstrap)
|
||||
{
|
||||
StitchWithConfidence(classifiedMat, _position, boosted: true, mode: mode);
|
||||
PaintExploredCircle(_position);
|
||||
if (_consecutiveMatchFails >= 30)
|
||||
{
|
||||
Log.Information("Re-bootstrap: mode={Mode} pos=({X:F1},{Y:F1}) frameSize={FS} walls={W} stitch={Ms:F1}ms",
|
||||
|
|
@ -118,6 +130,7 @@ public class WorldMap : IDisposable
|
|||
_position = matched;
|
||||
var stitchStart = sw.Elapsed.TotalMilliseconds;
|
||||
StitchWithConfidence(classifiedMat, _position, boosted: false, mode: mode);
|
||||
PaintExploredCircle(_position);
|
||||
var stitchMs = sw.Elapsed.TotalMilliseconds - stitchStart;
|
||||
|
||||
var posDx = _position.X - prevPos.X;
|
||||
|
|
@ -127,6 +140,41 @@ public class WorldMap : IDisposable
|
|||
return _position;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Noise filter + frame deduplication. Returns true if the frame should be skipped.
|
||||
/// </summary>
|
||||
private bool ShouldSkipFrame(Mat classifiedMat, Mat wallMask, bool isCorner, bool needsBootstrap, Stopwatch sw)
|
||||
{
|
||||
// Block-based noise filter: only needed for overlay (game effects bleed through)
|
||||
// Skip during warmup — we need walls to seed the canvas, confidence handles noise
|
||||
if (!isCorner && !needsBootstrap)
|
||||
{
|
||||
var cleanFraction = FilterNoisyBlocks(wallMask, classifiedMat);
|
||||
if (cleanFraction < 0.25)
|
||||
{
|
||||
Log.Information("Noise filter: {Clean:P0} clean, skipping ({Ms:F1}ms)",
|
||||
cleanFraction, sw.Elapsed.TotalMilliseconds);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
// Frame deduplication: skip if minimap hasn't scrolled yet (but always allow warmup through)
|
||||
if (!needsBootstrap && _prevWallMask != null && _frameCount > 1)
|
||||
{
|
||||
using var xor = new Mat();
|
||||
Cv2.BitwiseXor(wallMask, _prevWallMask, xor);
|
||||
var changedPixels = Cv2.CountNonZero(xor);
|
||||
if (changedPixels < _config.FrameChangeThreshold)
|
||||
{
|
||||
Log.Debug("Frame dedup: {Changed} changed pixels, skipping ({Ms:F1}ms)",
|
||||
changedPixels, sw.Elapsed.TotalMilliseconds);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
private MapPosition? MatchPosition(Mat wallMask, MapPosition estimate)
|
||||
{
|
||||
var frameSize = wallMask.Width;
|
||||
|
|
@ -143,8 +191,8 @@ public class WorldMap : IDisposable
|
|||
// Clamp to canvas bounds
|
||||
var sx0 = Math.Max(0, sx);
|
||||
var sy0 = Math.Max(0, sy);
|
||||
var sx1 = Math.Min(_config.CanvasSize, sx + searchSize);
|
||||
var sy1 = Math.Min(_config.CanvasSize, sy + searchSize);
|
||||
var sx1 = Math.Min(_canvasSize, sx + searchSize);
|
||||
var sy1 = Math.Min(_canvasSize, sy + searchSize);
|
||||
var sw = sx1 - sx0;
|
||||
var sh = sy1 - sy0;
|
||||
|
||||
|
|
@ -213,8 +261,8 @@ public class WorldMap : IDisposable
|
|||
var srcY = Math.Max(0, -canvasY);
|
||||
var dstX = Math.Max(0, canvasX);
|
||||
var dstY = Math.Max(0, canvasY);
|
||||
var w = Math.Min(frameSize - srcX, _config.CanvasSize - dstX);
|
||||
var h = Math.Min(frameSize - srcY, _config.CanvasSize - dstY);
|
||||
var w = Math.Min(frameSize - srcX, _canvasSize - dstX);
|
||||
var h = Math.Min(frameSize - srcY, _canvasSize - dstY);
|
||||
|
||||
if (w <= 0 || h <= 0) return;
|
||||
|
||||
|
|
@ -300,16 +348,23 @@ public class WorldMap : IDisposable
|
|||
}
|
||||
}
|
||||
|
||||
// Mark explored area: circle around player, overwrite Unknown and Fog
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Mark explored area: circle around player position, overwrite Unknown and Fog cells.
|
||||
/// Called by MatchAndStitch after stitch in both bootstrap and match-success branches.
|
||||
/// </summary>
|
||||
private void PaintExploredCircle(MapPosition position)
|
||||
{
|
||||
var pcx = (int)Math.Round(position.X);
|
||||
var pcy = (int)Math.Round(position.Y);
|
||||
var r = _config.ExploredRadius;
|
||||
var r2 = r * r;
|
||||
|
||||
var y0 = Math.Max(0, pcy - r);
|
||||
var y1 = Math.Min(_config.CanvasSize - 1, pcy + r);
|
||||
var y1 = Math.Min(_canvasSize - 1, pcy + r);
|
||||
var x0 = Math.Max(0, pcx - r);
|
||||
var x1 = Math.Min(_config.CanvasSize - 1, pcx + r);
|
||||
var x1 = Math.Min(_canvasSize - 1, pcx + r);
|
||||
|
||||
for (var y = y0; y <= y1; y++)
|
||||
for (var x = x0; x <= x1; x++)
|
||||
|
|
@ -363,140 +418,8 @@ public class WorldMap : IDisposable
|
|||
return totalBlocks > 0 ? (double)cleanBlocks / totalBlocks : 1.0;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// BFS through walkable (Explored) cells to find the best frontier direction.
|
||||
/// Instead of stopping at the nearest frontier, runs the full BFS and counts
|
||||
/// frontier cells reachable per first-step direction. Prefers directions with
|
||||
/// more frontier cells (corridors) over directions with few (dead-end rooms).
|
||||
/// </summary>
|
||||
public (double dirX, double dirY)? FindNearestUnexplored(MapPosition pos, int searchRadius = 400)
|
||||
{
|
||||
var cx = (int)Math.Round(pos.X);
|
||||
var cy = (int)Math.Round(pos.Y);
|
||||
|
||||
// BFS at half resolution for speed (step=2 → ~200x200 effective grid for r=400)
|
||||
const int step = 2;
|
||||
var size = _config.CanvasSize;
|
||||
var rr = searchRadius / step;
|
||||
var gridW = 2 * rr + 1;
|
||||
|
||||
var visited = new bool[gridW * gridW];
|
||||
// Propagate first step from start during BFS (avoids per-frontier trace-back)
|
||||
var firstStepX = new short[gridW * gridW];
|
||||
var firstStepY = new short[gridW * gridW];
|
||||
|
||||
var queue = new Queue<(int gx, int gy)>(4096);
|
||||
var startGx = rr;
|
||||
var startGy = rr;
|
||||
visited[startGy * gridW + startGx] = true;
|
||||
queue.Enqueue((startGx, startGy));
|
||||
|
||||
// 8-connected neighbors
|
||||
ReadOnlySpan<int> dxs = [-1, 0, 1, -1, 1, -1, 0, 1];
|
||||
ReadOnlySpan<int> dys = [-1, -1, -1, 0, 0, 1, 1, 1];
|
||||
|
||||
// Count frontier cells per first-step direction
|
||||
var frontierCounts = new Dictionary<int, int>();
|
||||
var firstStepCoords = new Dictionary<int, (short gx, short gy)>();
|
||||
|
||||
while (queue.Count > 0)
|
||||
{
|
||||
var (gx, gy) = queue.Dequeue();
|
||||
|
||||
// Map grid coords back to canvas coords
|
||||
var wx = cx + (gx - rr) * step;
|
||||
var wy = cy + (gy - rr) * step;
|
||||
|
||||
// Check if this explored cell borders Unknown/Fog (= frontier)
|
||||
if (gx != startGx || gy != startGy)
|
||||
{
|
||||
var cellIdx = gy * gridW + gx;
|
||||
for (var d = 0; d < 8; d++)
|
||||
{
|
||||
var nx = wx + dxs[d] * step;
|
||||
var ny = wy + dys[d] * step;
|
||||
if (nx < 0 || nx >= size || ny < 0 || ny >= size) continue;
|
||||
var neighbor = _canvas.At<byte>(ny, nx);
|
||||
if (neighbor == (byte)MapCell.Unknown || neighbor == (byte)MapCell.Fog)
|
||||
{
|
||||
var fsKey = firstStepY[cellIdx] * gridW + firstStepX[cellIdx];
|
||||
if (frontierCounts.TryGetValue(fsKey, out var cnt))
|
||||
frontierCounts[fsKey] = cnt + 1;
|
||||
else
|
||||
{
|
||||
frontierCounts[fsKey] = 1;
|
||||
firstStepCoords[fsKey] = (firstStepX[cellIdx], firstStepY[cellIdx]);
|
||||
}
|
||||
break; // don't double-count this cell
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Expand to walkable neighbors
|
||||
for (var d = 0; d < 8; d++)
|
||||
{
|
||||
var ngx = gx + dxs[d];
|
||||
var ngy = gy + dys[d];
|
||||
if (ngx < 0 || ngx >= gridW || ngy < 0 || ngy >= gridW) continue;
|
||||
|
||||
var idx = ngy * gridW + ngx;
|
||||
if (visited[idx]) continue;
|
||||
|
||||
var nwx = cx + (ngx - rr) * step;
|
||||
var nwy = cy + (ngy - rr) * step;
|
||||
if (nwx < 0 || nwx >= size || nwy < 0 || nwy >= size) continue;
|
||||
|
||||
var cell = _canvas.At<byte>(nwy, nwx);
|
||||
if (cell != (byte)MapCell.Explored) continue;
|
||||
|
||||
visited[idx] = true;
|
||||
// Propagate first step: direct neighbors of start ARE the first step
|
||||
if (gx == startGx && gy == startGy)
|
||||
{
|
||||
firstStepX[idx] = (short)ngx;
|
||||
firstStepY[idx] = (short)ngy;
|
||||
}
|
||||
else
|
||||
{
|
||||
var parentIdx = gy * gridW + gx;
|
||||
firstStepX[idx] = firstStepX[parentIdx];
|
||||
firstStepY[idx] = firstStepY[parentIdx];
|
||||
}
|
||||
queue.Enqueue((ngx, ngy));
|
||||
}
|
||||
}
|
||||
|
||||
if (frontierCounts.Count == 0)
|
||||
{
|
||||
Log.Information("BFS: no reachable frontier within {Radius}px", searchRadius);
|
||||
return null;
|
||||
}
|
||||
|
||||
// Pick direction with the most frontier cells (prefers corridors over dead ends)
|
||||
var bestKey = -1;
|
||||
var bestCount = 0;
|
||||
foreach (var (key, count) in frontierCounts)
|
||||
{
|
||||
if (count > bestCount)
|
||||
{
|
||||
bestCount = count;
|
||||
bestKey = key;
|
||||
}
|
||||
}
|
||||
|
||||
var (bestGx, bestGy) = firstStepCoords[bestKey];
|
||||
var dirX = (double)(bestGx - startGx);
|
||||
var dirY = (double)(bestGy - startGy);
|
||||
var len = Math.Sqrt(dirX * dirX + dirY * dirY);
|
||||
if (len < 0.001) return (1, 0);
|
||||
|
||||
dirX /= len;
|
||||
dirY /= len;
|
||||
|
||||
Log.Debug("BFS: {DirCount} directions, best={Best} frontier cells, dir=({Dx:F2},{Dy:F2})",
|
||||
frontierCounts.Count, bestCount, dirX, dirY);
|
||||
return (dirX, dirY);
|
||||
}
|
||||
=> _pathFinder.FindNearestUnexplored(_canvas, _canvasSize, pos, searchRadius);
|
||||
|
||||
public byte[] GetMapSnapshot()
|
||||
{
|
||||
|
|
@ -510,8 +433,8 @@ public class WorldMap : IDisposable
|
|||
var cy = (int)Math.Round(center.Y);
|
||||
var half = viewSize / 2;
|
||||
|
||||
var x0 = Math.Clamp(cx - half, 0, _config.CanvasSize - viewSize);
|
||||
var y0 = Math.Clamp(cy - half, 0, _config.CanvasSize - viewSize);
|
||||
var x0 = Math.Clamp(cx - half, 0, _canvasSize - viewSize);
|
||||
var y0 = Math.Clamp(cy - half, 0, _canvasSize - viewSize);
|
||||
var roi = new Mat(_canvas, new Rect(x0, y0, viewSize, viewSize));
|
||||
|
||||
using var colored = new Mat(viewSize, viewSize, MatType.CV_8UC3, new Scalar(23, 17, 13));
|
||||
|
|
@ -538,11 +461,14 @@ public class WorldMap : IDisposable
|
|||
|
||||
public void Reset()
|
||||
{
|
||||
_canvas.SetTo(Scalar.Black);
|
||||
_confidence.SetTo(Scalar.Black);
|
||||
_canvas.Dispose();
|
||||
_confidence.Dispose();
|
||||
_canvasSize = _config.CanvasSize;
|
||||
_canvas = new Mat(_canvasSize, _canvasSize, MatType.CV_8UC1, Scalar.Black);
|
||||
_confidence = new Mat(_canvasSize, _canvasSize, MatType.CV_16SC1, Scalar.Black);
|
||||
_prevWallMask?.Dispose();
|
||||
_prevWallMask = null;
|
||||
_position = new MapPosition(_config.CanvasSize / 2.0, _config.CanvasSize / 2.0);
|
||||
_position = new MapPosition(_canvasSize / 2.0, _canvasSize / 2.0);
|
||||
_frameCount = 0;
|
||||
_consecutiveMatchFails = 0;
|
||||
}
|
||||
|
|
@ -565,8 +491,8 @@ public class WorldMap : IDisposable
|
|||
var cy = (int)Math.Round(_position.Y);
|
||||
var x0 = Math.Max(0, cx - halfClear);
|
||||
var y0 = Math.Max(0, cy - halfClear);
|
||||
var w = Math.Min(_config.CanvasSize, cx + halfClear) - x0;
|
||||
var h = Math.Min(_config.CanvasSize, cy + halfClear) - y0;
|
||||
var w = Math.Min(_canvasSize, cx + halfClear) - x0;
|
||||
var h = Math.Min(_canvasSize, cy + halfClear) - y0;
|
||||
if (w > 0 && h > 0)
|
||||
{
|
||||
var rect = new Rect(x0, y0, w, h);
|
||||
|
|
|
|||
12
src/Poe2Trade.Screen/DetectionTypes.cs
Normal file
12
src/Poe2Trade.Screen/DetectionTypes.cs
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
namespace Poe2Trade.Screen;
|
||||
|
||||
public record DetectedEnemy(
|
||||
float Confidence,
|
||||
int X, int Y, int Width, int Height,
|
||||
int Cx, int Cy,
|
||||
bool HealthBarConfirmed);
|
||||
|
||||
public record DetectionSnapshot(
|
||||
IReadOnlyList<DetectedEnemy> Enemies,
|
||||
long Timestamp,
|
||||
float InferenceMs);
|
||||
|
|
@ -10,23 +10,30 @@ class DiffCropHandler
|
|||
{
|
||||
private Bitmap? _referenceFrame;
|
||||
private Region? _referenceRegion;
|
||||
private readonly object _refLock = new();
|
||||
|
||||
public void HandleSnapshot(string? file = null, Region? region = null)
|
||||
{
|
||||
var newFrame = ScreenCapture.CaptureOrLoad(file, region);
|
||||
lock (_refLock)
|
||||
{
|
||||
_referenceFrame?.Dispose();
|
||||
_referenceFrame = ScreenCapture.CaptureOrLoad(file, region);
|
||||
_referenceFrame = newFrame;
|
||||
_referenceRegion = region;
|
||||
}
|
||||
}
|
||||
|
||||
public void HandleScreenshot(string path, Region? region = null)
|
||||
{
|
||||
var bitmap = _referenceFrame ?? ScreenCapture.CaptureOrLoad(null, region);
|
||||
Bitmap? refCopy;
|
||||
lock (_refLock) { refCopy = _referenceFrame != null ? (Bitmap)_referenceFrame.Clone() : null; }
|
||||
var bitmap = refCopy ?? ScreenCapture.CaptureOrLoad(null, region);
|
||||
var format = ImageUtils.GetImageFormat(path);
|
||||
var dir = Path.GetDirectoryName(path);
|
||||
if (!string.IsNullOrEmpty(dir) && !Directory.Exists(dir))
|
||||
Directory.CreateDirectory(dir);
|
||||
bitmap.Save(path, format);
|
||||
if (bitmap != _referenceFrame) bitmap.Dispose();
|
||||
bitmap.Dispose();
|
||||
}
|
||||
|
||||
public byte[] HandleCapture(Region? region = null)
|
||||
|
|
@ -43,48 +50,55 @@ class DiffCropHandler
|
|||
/// </summary>
|
||||
public (Bitmap cropped, Bitmap refCropped, Bitmap current, Region region)? DiffCrop(
|
||||
DiffCropParams c, string? file = null, Region? region = null)
|
||||
{
|
||||
Bitmap refSnapshot;
|
||||
Region? refRegion;
|
||||
lock (_refLock)
|
||||
{
|
||||
if (_referenceFrame == null)
|
||||
return null;
|
||||
refSnapshot = (Bitmap)_referenceFrame.Clone();
|
||||
refRegion = _referenceRegion;
|
||||
}
|
||||
|
||||
var diffRegion = region ?? _referenceRegion;
|
||||
var diffRegion = region ?? refRegion;
|
||||
int baseX = diffRegion?.X ?? 0;
|
||||
int baseY = diffRegion?.Y ?? 0;
|
||||
var current = ScreenCapture.CaptureOrLoad(file, diffRegion);
|
||||
|
||||
Bitmap refForDiff = _referenceFrame;
|
||||
bool disposeRef = false;
|
||||
Bitmap refForDiff = refSnapshot;
|
||||
|
||||
if (diffRegion != null)
|
||||
{
|
||||
if (_referenceRegion == null)
|
||||
if (refRegion == null)
|
||||
{
|
||||
var croppedRef = CropBitmap(_referenceFrame, diffRegion);
|
||||
var croppedRef = CropBitmap(refSnapshot, diffRegion);
|
||||
if (croppedRef == null)
|
||||
{
|
||||
current.Dispose();
|
||||
refSnapshot.Dispose();
|
||||
return null;
|
||||
}
|
||||
refForDiff = croppedRef;
|
||||
disposeRef = true;
|
||||
}
|
||||
else if (!RegionsEqual(diffRegion, _referenceRegion))
|
||||
else if (!RegionsEqual(diffRegion, refRegion))
|
||||
{
|
||||
int offX = diffRegion.X - _referenceRegion.X;
|
||||
int offY = diffRegion.Y - _referenceRegion.Y;
|
||||
if (offX < 0 || offY < 0 || offX + diffRegion.Width > _referenceFrame.Width || offY + diffRegion.Height > _referenceFrame.Height)
|
||||
int offX = diffRegion.X - refRegion.X;
|
||||
int offY = diffRegion.Y - refRegion.Y;
|
||||
if (offX < 0 || offY < 0 || offX + diffRegion.Width > refSnapshot.Width || offY + diffRegion.Height > refSnapshot.Height)
|
||||
{
|
||||
current.Dispose();
|
||||
refSnapshot.Dispose();
|
||||
return null;
|
||||
}
|
||||
var croppedRef = CropBitmap(_referenceFrame, new Region(offX, offY, diffRegion.Width, diffRegion.Height));
|
||||
var croppedRef = CropBitmap(refSnapshot, new Region(offX, offY, diffRegion.Width, diffRegion.Height));
|
||||
if (croppedRef == null)
|
||||
{
|
||||
current.Dispose();
|
||||
refSnapshot.Dispose();
|
||||
return null;
|
||||
}
|
||||
refForDiff = croppedRef;
|
||||
disposeRef = true;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -126,7 +140,8 @@ class DiffCropHandler
|
|||
if (totalChanged == 0)
|
||||
{
|
||||
current.Dispose();
|
||||
if (disposeRef) refForDiff.Dispose();
|
||||
if (refForDiff != refSnapshot) refForDiff.Dispose();
|
||||
refSnapshot.Dispose();
|
||||
return null;
|
||||
}
|
||||
|
||||
|
|
@ -226,7 +241,8 @@ class DiffCropHandler
|
|||
{
|
||||
Log.Debug("diff-crop: no tooltip-sized region found");
|
||||
current.Dispose();
|
||||
if (disposeRef) refForDiff.Dispose();
|
||||
if (refForDiff != refSnapshot) refForDiff.Dispose();
|
||||
refSnapshot.Dispose();
|
||||
return null;
|
||||
}
|
||||
|
||||
|
|
@ -307,7 +323,8 @@ class DiffCropHandler
|
|||
|
||||
Log.Debug("diff-crop: tooltip region ({X},{Y}) {W}x{H}", minX, minY, rw, rh);
|
||||
|
||||
if (disposeRef) refForDiff.Dispose();
|
||||
if (refForDiff != refSnapshot) refForDiff.Dispose();
|
||||
refSnapshot.Dispose();
|
||||
return (cropped, refCropped, current, resultRegion);
|
||||
}
|
||||
|
||||
|
|
|
|||
173
src/Poe2Trade.Screen/EnemyDetector.cs
Normal file
173
src/Poe2Trade.Screen/EnemyDetector.cs
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
using OpenCvSharp;
|
||||
using Poe2Trade.Core;
|
||||
using Serilog;
|
||||
using Region = Poe2Trade.Core.Region;
|
||||
|
||||
namespace Poe2Trade.Screen;
|
||||
|
||||
/// <summary>
|
||||
/// Detects enemies on screen using two-stage approach:
|
||||
/// 1. YOLO detection via Python daemon (~5Hz, every 6th frame)
|
||||
/// 2. Health bar confirmation via HSV threshold on bbox region (every frame, ~1ms)
|
||||
/// </summary>
|
||||
public class EnemyDetector : IFrameConsumer, IDisposable
|
||||
{
|
||||
// Crop region for gameplay area at 2560x1440 — excludes HUD globes, minimap
|
||||
private static readonly Region GameplayRegion = new(320, 100, 1920, 1200);
|
||||
|
||||
private const int DetectEveryNFrames = 6; // ~5Hz at 30fps
|
||||
private const int HealthBarHeight = 10; // px above bbox to scan for red bar
|
||||
private const float RedPixelThreshold = 0.05f; // 5% red pixels = confirmed
|
||||
|
||||
private readonly PythonDetectBridge _bridge = new();
|
||||
private volatile DetectionSnapshot _latest = new([], 0, 0);
|
||||
private int _frameCounter;
|
||||
private List<DetectedEnemy> _activeEnemies = [];
|
||||
|
||||
public bool Enabled { get; set; }
|
||||
public DetectionSnapshot Latest => _latest;
|
||||
public event Action<DetectionSnapshot>? DetectionUpdated;
|
||||
|
||||
public void Process(ScreenFrame frame)
|
||||
{
|
||||
if (!Enabled) return;
|
||||
|
||||
_frameCounter++;
|
||||
|
||||
// Health bar confirmation runs every frame for known enemies
|
||||
if (_activeEnemies.Count > 0)
|
||||
{
|
||||
_activeEnemies = ConfirmHealthBars(frame, _activeEnemies);
|
||||
}
|
||||
|
||||
// YOLO detection runs every Nth frame
|
||||
if (_frameCounter % DetectEveryNFrames != 0) return;
|
||||
|
||||
try
|
||||
{
|
||||
// Bounds check
|
||||
if (GameplayRegion.X + GameplayRegion.Width > frame.Width ||
|
||||
GameplayRegion.Y + GameplayRegion.Height > frame.Height)
|
||||
return;
|
||||
|
||||
using var cropped = frame.CropBgr(GameplayRegion);
|
||||
var result = _bridge.Detect(cropped);
|
||||
|
||||
// Offset bbox coords by crop origin → screen-space coordinates
|
||||
var enemies = new List<DetectedEnemy>(result.Count);
|
||||
foreach (var det in result.Detections)
|
||||
{
|
||||
var screenX = det.X + GameplayRegion.X;
|
||||
var screenY = det.Y + GameplayRegion.Y;
|
||||
var screenCx = det.Cx + GameplayRegion.X;
|
||||
var screenCy = det.Cy + GameplayRegion.Y;
|
||||
|
||||
// Check if this enemy was previously confirmed
|
||||
var wasConfirmed = _activeEnemies.Any(e =>
|
||||
Math.Abs(e.Cx - screenCx) < 50 && Math.Abs(e.Cy - screenCy) < 50 &&
|
||||
e.HealthBarConfirmed);
|
||||
|
||||
enemies.Add(new DetectedEnemy(
|
||||
det.Confidence,
|
||||
screenX, screenY, det.Width, det.Height,
|
||||
screenCx, screenCy,
|
||||
wasConfirmed));
|
||||
}
|
||||
|
||||
_activeEnemies = enemies;
|
||||
|
||||
var snapshot = new DetectionSnapshot(
|
||||
enemies.AsReadOnly(),
|
||||
DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
|
||||
result.InferenceMs);
|
||||
|
||||
_latest = snapshot;
|
||||
DetectionUpdated?.Invoke(snapshot);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Log.Debug(ex, "EnemyDetector YOLO failed");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Scan a narrow band above each enemy bbox for red health bar pixels (HSV threshold).
|
||||
/// Returns updated list with HealthBarConfirmed set where detected.
|
||||
/// </summary>
|
||||
private static List<DetectedEnemy> ConfirmHealthBars(ScreenFrame frame, List<DetectedEnemy> enemies)
|
||||
{
|
||||
var updated = new List<DetectedEnemy>(enemies.Count);
|
||||
|
||||
foreach (var enemy in enemies)
|
||||
{
|
||||
if (enemy.HealthBarConfirmed)
|
||||
{
|
||||
updated.Add(enemy);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Scan region: narrow strip above the bbox top edge
|
||||
var scanY = Math.Max(0, enemy.Y - HealthBarHeight);
|
||||
var scanHeight = Math.Min(HealthBarHeight, enemy.Y);
|
||||
if (scanHeight <= 0 || enemy.Width <= 0)
|
||||
{
|
||||
updated.Add(enemy);
|
||||
continue;
|
||||
}
|
||||
|
||||
var scanRegion = new Region(
|
||||
Math.Max(0, enemy.X),
|
||||
scanY,
|
||||
Math.Min(enemy.Width, frame.Width - Math.Max(0, enemy.X)),
|
||||
scanHeight);
|
||||
|
||||
if (scanRegion.Width <= 0 || scanRegion.Height <= 0 ||
|
||||
scanRegion.X + scanRegion.Width > frame.Width ||
|
||||
scanRegion.Y + scanRegion.Height > frame.Height)
|
||||
{
|
||||
updated.Add(enemy);
|
||||
continue;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var confirmed = HasRedHealthBar(frame, scanRegion);
|
||||
updated.Add(enemy with { HealthBarConfirmed = confirmed });
|
||||
}
|
||||
catch
|
||||
{
|
||||
updated.Add(enemy);
|
||||
}
|
||||
}
|
||||
|
||||
return updated;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Check if a region contains enough red pixels to indicate a health bar.
|
||||
/// Red in HSV: H=0-10 or H=170-180, S>100, V>80.
|
||||
/// </summary>
|
||||
private static bool HasRedHealthBar(ScreenFrame frame, Region region)
|
||||
{
|
||||
using var bgr = frame.CropBgr(region);
|
||||
using var hsv = new Mat();
|
||||
Cv2.CvtColor(bgr, hsv, ColorConversionCodes.BGR2HSV);
|
||||
|
||||
// Red wraps around in HSV — check both ranges
|
||||
using var mask1 = new Mat();
|
||||
using var mask2 = new Mat();
|
||||
Cv2.InRange(hsv, new Scalar(0, 100, 80), new Scalar(10, 255, 255), mask1);
|
||||
Cv2.InRange(hsv, new Scalar(170, 100, 80), new Scalar(180, 255, 255), mask2);
|
||||
|
||||
using var combined = new Mat();
|
||||
Cv2.BitwiseOr(mask1, mask2, combined);
|
||||
|
||||
var totalPixels = combined.Rows * combined.Cols;
|
||||
var redPixels = Cv2.CountNonZero(combined);
|
||||
var ratio = (float)redPixels / totalPixels;
|
||||
|
||||
return ratio >= RedPixelThreshold;
|
||||
}
|
||||
|
||||
public void Dispose() => _bridge.Dispose();
|
||||
}
|
||||
|
|
@ -13,6 +13,7 @@ public class FramePipeline : IDisposable
|
|||
public IScreenCapture Capture => _capture;
|
||||
|
||||
public void AddConsumer(IFrameConsumer consumer) => _consumers.Add(consumer);
|
||||
public void RemoveConsumer(IFrameConsumer consumer) => _consumers.Remove(consumer);
|
||||
|
||||
/// <summary>
|
||||
/// Capture one frame, dispatch to all consumers in parallel, then dispose frame.
|
||||
|
|
|
|||
37
src/Poe2Trade.Screen/FramePipelineService.cs
Normal file
37
src/Poe2Trade.Screen/FramePipelineService.cs
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
using Serilog;
|
||||
|
||||
namespace Poe2Trade.Screen;
|
||||
|
||||
public class FramePipelineService : IDisposable
|
||||
{
|
||||
public FramePipeline Pipeline { get; }
|
||||
public IScreenCapture Backend { get; }
|
||||
|
||||
public FramePipelineService()
|
||||
{
|
||||
Backend = CreateBackend();
|
||||
Pipeline = new FramePipeline(Backend);
|
||||
}
|
||||
|
||||
private static IScreenCapture CreateBackend()
|
||||
{
|
||||
try
|
||||
{
|
||||
var dxgi = new DesktopDuplication();
|
||||
Log.Information("Screen capture: DXGI Desktop Duplication");
|
||||
return dxgi;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Log.Warning(ex, "DXGI unavailable, falling back to GDI");
|
||||
}
|
||||
|
||||
Log.Information("Screen capture: GDI (CopyFromScreen)");
|
||||
return new GdiCapture();
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
Pipeline.Dispose();
|
||||
}
|
||||
}
|
||||
118
src/Poe2Trade.Screen/FrameSaver.cs
Normal file
118
src/Poe2Trade.Screen/FrameSaver.cs
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
using OpenCvSharp;
|
||||
using Serilog;
|
||||
using Region = Poe2Trade.Core.Region;
|
||||
|
||||
namespace Poe2Trade.Screen;
|
||||
|
||||
/// <summary>
|
||||
/// Saves full-screen frames as JPEGs for YOLO training data collection.
|
||||
/// Only saves when: (1) health bars detected, (2) scene has changed since last save.
|
||||
/// This avoids flooding disk when standing still in a dense pack.
|
||||
/// </summary>
|
||||
public class FrameSaver : IFrameConsumer
|
||||
{
|
||||
// Gameplay area at 2560x1440 — excludes HUD globes, minimap
|
||||
private static readonly Region GameplayRegion = new(320, 100, 1920, 1200);
|
||||
|
||||
private const int JpegQuality = 95;
|
||||
private const int MinSaveIntervalMs = 1000;
|
||||
private const int MinRedPixels = 50;
|
||||
private const int ThumbSize = 64;
|
||||
private const double MovementThreshold = 8.0; // mean absolute diff on 64x64 grayscale
|
||||
|
||||
private readonly string _outputDir;
|
||||
private int _savedCount;
|
||||
private long _lastSaveTime;
|
||||
private Mat? _prevThumb;
|
||||
|
||||
public bool Enabled { get; set; }
|
||||
public int SavedCount => _savedCount;
|
||||
|
||||
public FrameSaver(string outputDir = "training-data/raw")
|
||||
{
|
||||
_outputDir = Path.GetFullPath(outputDir);
|
||||
}
|
||||
|
||||
public void Process(ScreenFrame frame)
|
||||
{
|
||||
if (!Enabled) return;
|
||||
|
||||
var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
|
||||
if (now - _lastSaveTime < MinSaveIntervalMs) return;
|
||||
|
||||
if (GameplayRegion.X + GameplayRegion.Width > frame.Width ||
|
||||
GameplayRegion.Y + GameplayRegion.Height > frame.Height)
|
||||
return;
|
||||
|
||||
try
|
||||
{
|
||||
using var bgr = frame.CropBgr(GameplayRegion);
|
||||
|
||||
if (!HasHealthBars(bgr)) return;
|
||||
if (!HasSceneChanged(bgr)) return;
|
||||
|
||||
if (!Directory.Exists(_outputDir))
|
||||
Directory.CreateDirectory(_outputDir);
|
||||
|
||||
var fullRegion = new Region(0, 0, frame.Width, frame.Height);
|
||||
using var fullBgr = frame.CropBgr(fullRegion);
|
||||
|
||||
var path = Path.Combine(_outputDir, $"frame_{now}.jpg");
|
||||
var prms = new ImageEncodingParam(ImwriteFlags.JpegQuality, JpegQuality);
|
||||
Cv2.ImWrite(path, fullBgr, [prms]);
|
||||
|
||||
_savedCount++;
|
||||
_lastSaveTime = now;
|
||||
|
||||
if (_savedCount % 10 == 0)
|
||||
Log.Information("FrameSaver: saved {Count} frames to {Dir}", _savedCount, _outputDir);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Log.Debug(ex, "FrameSaver failed to save frame");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Scan for enemy health bar pixels (HSV threshold).
|
||||
/// Target colors: #C1251E, #BB281C → H≈1-3, S≈215, V≈187-193.
|
||||
/// </summary>
|
||||
private static bool HasHealthBars(Mat bgr)
|
||||
{
|
||||
using var hsv = new Mat();
|
||||
Cv2.CvtColor(bgr, hsv, ColorConversionCodes.BGR2HSV);
|
||||
|
||||
using var mask = new Mat();
|
||||
Cv2.InRange(hsv, new Scalar(0, 150, 130), new Scalar(8, 255, 255), mask);
|
||||
|
||||
return Cv2.CountNonZero(mask) >= MinRedPixels;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Compare a 64x64 grayscale thumbnail to the previous save.
|
||||
/// Returns true if the scene changed enough (character moved).
|
||||
/// </summary>
|
||||
private bool HasSceneChanged(Mat bgr)
|
||||
{
|
||||
using var gray = new Mat();
|
||||
Cv2.CvtColor(bgr, gray, ColorConversionCodes.BGR2GRAY);
|
||||
|
||||
var thumb = new Mat();
|
||||
Cv2.Resize(gray, thumb, new Size(ThumbSize, ThumbSize), interpolation: InterpolationFlags.Area);
|
||||
|
||||
if (_prevThumb == null)
|
||||
{
|
||||
_prevThumb = thumb;
|
||||
return true; // first frame, always save
|
||||
}
|
||||
|
||||
using var diff = new Mat();
|
||||
Cv2.Absdiff(thumb, _prevThumb, diff);
|
||||
var mad = Cv2.Mean(diff).Val0;
|
||||
|
||||
_prevThumb.Dispose();
|
||||
_prevThumb = thumb;
|
||||
|
||||
return mad >= MovementThreshold;
|
||||
}
|
||||
}
|
||||
35
src/Poe2Trade.Screen/GameStateDetector.cs
Normal file
35
src/Poe2Trade.Screen/GameStateDetector.cs
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
using Poe2Trade.Core;
|
||||
using Serilog;
|
||||
|
||||
namespace Poe2Trade.Screen;
|
||||
|
||||
/// <summary>
|
||||
/// Classifies the current game UI state by probing known pixel positions on each frame.
|
||||
/// Near-zero cost (~0.01ms per frame) — just a few pixel reads.
|
||||
/// </summary>
|
||||
public class GameStateDetector : IFrameConsumer, IGameStateProvider
|
||||
{
|
||||
private volatile GameUiState _currentState = GameUiState.Unknown;
|
||||
|
||||
public GameUiState CurrentState => _currentState;
|
||||
public event Action<GameUiState, GameUiState>? StateChanged;
|
||||
|
||||
public void Process(ScreenFrame frame)
|
||||
{
|
||||
var newState = Classify(frame);
|
||||
var old = _currentState;
|
||||
if (newState == old) return;
|
||||
|
||||
_currentState = newState;
|
||||
Log.Debug("GameState: {Old} → {New}", old, newState);
|
||||
StateChanged?.Invoke(old, newState);
|
||||
}
|
||||
|
||||
private GameUiState Classify(ScreenFrame frame)
|
||||
{
|
||||
// TODO: Calibrate pixel probe positions from actual 2560x1440 screenshots.
|
||||
// Each state has 2-3 characteristic pixels that distinguish it.
|
||||
// For now, return Unknown — actual detection requires screenshot calibration.
|
||||
return GameUiState.Unknown;
|
||||
}
|
||||
}
|
||||
110
src/Poe2Trade.Screen/HudReader.cs
Normal file
110
src/Poe2Trade.Screen/HudReader.cs
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
using System.Drawing;
|
||||
using System.Text.RegularExpressions;
|
||||
using OpenCvSharp;
|
||||
using Poe2Trade.Core;
|
||||
using Serilog;
|
||||
using Region = Poe2Trade.Core.Region;
|
||||
|
||||
namespace Poe2Trade.Screen;
|
||||
|
||||
public record HudValues(int Current, int Max);
|
||||
|
||||
public record HudSnapshot
|
||||
{
|
||||
public HudValues? Life { get; init; }
|
||||
public HudValues? Mana { get; init; }
|
||||
public HudValues? EnergyShield { get; init; }
|
||||
public HudValues? Spirit { get; init; }
|
||||
public long Timestamp { get; init; }
|
||||
|
||||
public float LifePct => Life is { Max: > 0 } l ? (float)l.Current / l.Max : 1f;
|
||||
public float ManaPct => Mana is { Max: > 0 } m ? (float)m.Current / m.Max : 1f;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads life/mana/ES/spirit values from HUD globe text via OCR.
|
||||
/// Throttled to ~1 read per second (every 30 frames at 30fps).
|
||||
/// </summary>
|
||||
public class HudReader : IFrameConsumer
|
||||
{
|
||||
private static readonly Regex ValuePattern = new(@"(\d+)\s*/\s*(\d+)", RegexOptions.Compiled);
|
||||
|
||||
// Crop regions for HUD text at 2560x1440 — placeholders, need calibration
|
||||
private static readonly Region LifeRegion = new(100, 1340, 200, 40);
|
||||
private static readonly Region ManaRegion = new(2260, 1340, 200, 40);
|
||||
private static readonly Region EsRegion = new(100, 1300, 200, 40);
|
||||
private static readonly Region SpiritRegion = new(2260, 1300, 200, 40);
|
||||
|
||||
private const int OcrEveryNFrames = 30;
|
||||
|
||||
private readonly PythonOcrBridge _ocr = new();
|
||||
private volatile HudSnapshot _current = new() { Timestamp = 0 };
|
||||
private int _frameCounter;
|
||||
|
||||
public HudSnapshot Current => _current;
|
||||
public event Action<HudSnapshot>? Updated;
|
||||
public event Action<HudSnapshot>? LowLife;
|
||||
|
||||
public void Process(ScreenFrame frame)
|
||||
{
|
||||
if (++_frameCounter % OcrEveryNFrames != 0) return;
|
||||
|
||||
try
|
||||
{
|
||||
var life = ReadValue(frame, LifeRegion);
|
||||
var mana = ReadValue(frame, ManaRegion);
|
||||
var es = ReadValue(frame, EsRegion);
|
||||
var spirit = ReadValue(frame, SpiritRegion);
|
||||
|
||||
var snapshot = new HudSnapshot
|
||||
{
|
||||
Life = life,
|
||||
Mana = mana,
|
||||
EnergyShield = es,
|
||||
Spirit = spirit,
|
||||
Timestamp = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
|
||||
};
|
||||
|
||||
_current = snapshot;
|
||||
Updated?.Invoke(snapshot);
|
||||
|
||||
if (snapshot.LifePct < 0.3f)
|
||||
LowLife?.Invoke(snapshot);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Log.Debug(ex, "HudReader OCR failed");
|
||||
}
|
||||
}
|
||||
|
||||
private HudValues? ReadValue(ScreenFrame frame, Region region)
|
||||
{
|
||||
// Bounds check
|
||||
if (region.X + region.Width > frame.Width || region.Y + region.Height > frame.Height)
|
||||
return null;
|
||||
|
||||
using var bgr = frame.CropBgr(region);
|
||||
using var gray = new Mat();
|
||||
Cv2.CvtColor(bgr, gray, ColorConversionCodes.BGR2GRAY);
|
||||
|
||||
// Threshold for white text on dark background
|
||||
using var thresh = new Mat();
|
||||
Cv2.Threshold(gray, thresh, 180, 255, ThresholdTypes.Binary);
|
||||
|
||||
// Convert to Bitmap for OCR bridge
|
||||
var bytes = thresh.ToBytes(".png");
|
||||
using var ms = new System.IO.MemoryStream(bytes);
|
||||
using var bitmap = new Bitmap(ms);
|
||||
|
||||
var result = _ocr.OcrFromBitmap(bitmap);
|
||||
if (string.IsNullOrWhiteSpace(result.Text)) return null;
|
||||
|
||||
var match = ValuePattern.Match(result.Text);
|
||||
if (!match.Success) return null;
|
||||
|
||||
return new HudValues(
|
||||
int.Parse(match.Groups[1].Value),
|
||||
int.Parse(match.Groups[2].Value)
|
||||
);
|
||||
}
|
||||
}
|
||||
195
src/Poe2Trade.Screen/PythonDetectBridge.cs
Normal file
195
src/Poe2Trade.Screen/PythonDetectBridge.cs
Normal file
|
|
@ -0,0 +1,195 @@
|
|||
namespace Poe2Trade.Screen;
|
||||
|
||||
using System.Diagnostics;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using OpenCvSharp;
|
||||
using Serilog;
|
||||
|
||||
/// <summary>
|
||||
/// Manages a persistent Python subprocess for YOLO object detection.
|
||||
/// Lazy-starts on first request; reuses the process for subsequent calls.
|
||||
/// Same stdin/stdout JSON-per-line protocol as PythonOcrBridge.
|
||||
/// </summary>
|
||||
class PythonDetectBridge : IDisposable
|
||||
{
|
||||
private static readonly JsonSerializerOptions JsonOptions = new()
|
||||
{
|
||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||
};
|
||||
|
||||
private Process? _proc;
|
||||
private readonly string _daemonScript;
|
||||
private readonly string _pythonExe;
|
||||
private readonly object _lock = new();
|
||||
|
||||
public PythonDetectBridge()
|
||||
{
|
||||
_daemonScript = Path.GetFullPath(Path.Combine("tools", "python-detect", "daemon.py"));
|
||||
|
||||
var venvPython = Path.GetFullPath(Path.Combine("tools", "python-detect", ".venv", "Scripts", "python.exe"));
|
||||
_pythonExe = File.Exists(venvPython) ? venvPython : "python";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Run YOLO detection on a BGR Mat. Returns parsed detection results.
|
||||
/// </summary>
|
||||
public DetectResponse Detect(Mat bgrMat, float conf = 0.3f, float iou = 0.45f, int imgsz = 640)
|
||||
{
|
||||
EnsureRunning();
|
||||
|
||||
var imageBytes = bgrMat.ToBytes(".png");
|
||||
var imageBase64 = Convert.ToBase64String(imageBytes);
|
||||
|
||||
var req = new Dictionary<string, object?>
|
||||
{
|
||||
["cmd"] = "detect",
|
||||
["imageBase64"] = imageBase64,
|
||||
["conf"] = conf,
|
||||
["iou"] = iou,
|
||||
["imgsz"] = imgsz,
|
||||
};
|
||||
|
||||
return SendRequest(req);
|
||||
}
|
||||
|
||||
private DetectResponse SendRequest(object req)
|
||||
{
|
||||
var json = JsonSerializer.Serialize(req, JsonOptions);
|
||||
|
||||
string responseLine;
|
||||
lock (_lock)
|
||||
{
|
||||
_proc!.StandardInput.WriteLine(json);
|
||||
_proc.StandardInput.Flush();
|
||||
responseLine = _proc.StandardOutput.ReadLine()
|
||||
?? throw new Exception("Python detect daemon returned null");
|
||||
}
|
||||
|
||||
var resp = JsonSerializer.Deserialize<PythonDetectResponse>(responseLine, JsonOptions);
|
||||
if (resp == null)
|
||||
throw new Exception("Failed to parse Python detect response");
|
||||
if (!resp.Ok)
|
||||
throw new Exception(resp.Error ?? "Python detect failed");
|
||||
|
||||
return new DetectResponse
|
||||
{
|
||||
Count = resp.Count,
|
||||
InferenceMs = resp.InferenceMs,
|
||||
Detections = resp.Detections ?? [],
|
||||
};
|
||||
}
|
||||
|
||||
private void EnsureRunning()
|
||||
{
|
||||
if (_proc != null && !_proc.HasExited)
|
||||
return;
|
||||
|
||||
_proc?.Dispose();
|
||||
_proc = null;
|
||||
|
||||
if (!File.Exists(_daemonScript))
|
||||
throw new Exception($"Python detect daemon not found at {_daemonScript}");
|
||||
|
||||
Log.Information("Spawning Python detect daemon: {Python} {Script}", _pythonExe, _daemonScript);
|
||||
|
||||
var proc = new Process
|
||||
{
|
||||
StartInfo = new ProcessStartInfo
|
||||
{
|
||||
FileName = _pythonExe,
|
||||
Arguments = $"\"{_daemonScript}\"",
|
||||
UseShellExecute = false,
|
||||
RedirectStandardInput = true,
|
||||
RedirectStandardOutput = true,
|
||||
RedirectStandardError = true,
|
||||
CreateNoWindow = true,
|
||||
}
|
||||
};
|
||||
|
||||
proc.ErrorDataReceived += (_, e) =>
|
||||
{
|
||||
if (!string.IsNullOrEmpty(e.Data))
|
||||
Log.Debug("[python-detect] {Line}", e.Data);
|
||||
};
|
||||
|
||||
try
|
||||
{
|
||||
proc.Start();
|
||||
proc.BeginErrorReadLine();
|
||||
|
||||
// Wait for ready signal (up to 60s for CUDA warmup)
|
||||
var readyTask = Task.Run(() => proc.StandardOutput.ReadLine());
|
||||
if (!readyTask.Wait(TimeSpan.FromSeconds(60)))
|
||||
throw new Exception("Python detect daemon timed out waiting for ready signal");
|
||||
|
||||
var readyLine = readyTask.Result;
|
||||
if (readyLine == null)
|
||||
throw new Exception("Python detect daemon exited before ready signal");
|
||||
|
||||
var ready = JsonSerializer.Deserialize<PythonDetectResponse>(readyLine, JsonOptions);
|
||||
if (ready?.Ready != true)
|
||||
throw new Exception($"Python detect daemon did not send ready signal: {readyLine}");
|
||||
}
|
||||
catch
|
||||
{
|
||||
try { if (!proc.HasExited) proc.Kill(); } catch { /* best effort */ }
|
||||
proc.Dispose();
|
||||
throw;
|
||||
}
|
||||
|
||||
_proc = proc;
|
||||
Log.Information("Python detect daemon ready");
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (_proc != null && !_proc.HasExited)
|
||||
{
|
||||
try
|
||||
{
|
||||
_proc.StandardInput.Close();
|
||||
_proc.WaitForExit(3000);
|
||||
if (!_proc.HasExited) _proc.Kill();
|
||||
}
|
||||
catch { /* ignore */ }
|
||||
}
|
||||
_proc?.Dispose();
|
||||
_proc = null;
|
||||
}
|
||||
|
||||
// -- Response types --
|
||||
|
||||
public class DetectResponse
|
||||
{
|
||||
public int Count { get; set; }
|
||||
public float InferenceMs { get; set; }
|
||||
public List<Detection> Detections { get; set; } = [];
|
||||
}
|
||||
|
||||
public class Detection
|
||||
{
|
||||
[JsonPropertyName("class")]
|
||||
public string ClassName { get; set; } = "";
|
||||
|
||||
public int ClassId { get; set; }
|
||||
public float Confidence { get; set; }
|
||||
public int X { get; set; }
|
||||
public int Y { get; set; }
|
||||
public int Width { get; set; }
|
||||
public int Height { get; set; }
|
||||
public int Cx { get; set; }
|
||||
public int Cy { get; set; }
|
||||
}
|
||||
|
||||
private class PythonDetectResponse
|
||||
{
|
||||
public bool Ok { get; set; }
|
||||
public bool? Ready { get; set; }
|
||||
public string? Error { get; set; }
|
||||
public int Count { get; set; }
|
||||
public float InferenceMs { get; set; }
|
||||
public List<Detection>? Detections { get; set; }
|
||||
}
|
||||
}
|
||||
|
|
@ -104,7 +104,7 @@ class PythonOcrBridge : IDisposable
|
|||
|
||||
Log.Information("Spawning Python OCR daemon: {Python} {Script}", _pythonExe, _daemonScript);
|
||||
|
||||
_proc = new Process
|
||||
var proc = new Process
|
||||
{
|
||||
StartInfo = new ProcessStartInfo
|
||||
{
|
||||
|
|
@ -118,24 +118,35 @@ class PythonOcrBridge : IDisposable
|
|||
}
|
||||
};
|
||||
|
||||
_proc.ErrorDataReceived += (_, e) =>
|
||||
proc.ErrorDataReceived += (_, e) =>
|
||||
{
|
||||
if (!string.IsNullOrEmpty(e.Data))
|
||||
Log.Debug("[python-ocr] {Line}", e.Data);
|
||||
};
|
||||
|
||||
_proc.Start();
|
||||
_proc.BeginErrorReadLine();
|
||||
try
|
||||
{
|
||||
proc.Start();
|
||||
proc.BeginErrorReadLine();
|
||||
|
||||
// Wait for ready signal (up to 30s for first model load)
|
||||
var readyLine = _proc.StandardOutput.ReadLine();
|
||||
var readyLine = proc.StandardOutput.ReadLine();
|
||||
if (readyLine == null)
|
||||
throw new Exception("Python OCR daemon exited before ready signal");
|
||||
|
||||
var ready = JsonSerializer.Deserialize<PythonResponse>(readyLine, JsonOptions);
|
||||
if (ready?.Ready != true)
|
||||
throw new Exception($"Python OCR daemon did not send ready signal: {readyLine}");
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Kill orphaned process before re-throwing
|
||||
try { if (!proc.HasExited) proc.Kill(); } catch { /* best effort */ }
|
||||
proc.Dispose();
|
||||
throw;
|
||||
}
|
||||
|
||||
_proc = proc;
|
||||
Log.Information("Python OCR daemon ready");
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -238,16 +238,16 @@ public class ScreenReader : IScreenReader
|
|||
private static double BigramSimilarity(string a, string b)
|
||||
{
|
||||
if (a.Length < 2 || b.Length < 2) return a == b ? 1 : 0;
|
||||
var bigramsA = new Dictionary<string, int>();
|
||||
var bigramsA = new Dictionary<(char, char), int>();
|
||||
for (var i = 0; i < a.Length - 1; i++)
|
||||
{
|
||||
var bg = a.Substring(i, 2);
|
||||
var bg = (a[i], a[i + 1]);
|
||||
bigramsA[bg] = bigramsA.GetValueOrDefault(bg) + 1;
|
||||
}
|
||||
var matches = 0;
|
||||
for (var i = 0; i < b.Length - 1; i++)
|
||||
{
|
||||
var bg = b.Substring(i, 2);
|
||||
var bg = (b[i], b[i + 1]);
|
||||
if (bigramsA.TryGetValue(bg, out var count) && count > 0)
|
||||
{
|
||||
matches++;
|
||||
|
|
|
|||
|
|
@ -40,6 +40,7 @@ public partial class App : Application
|
|||
services.AddSingleton<IInventoryManager, InventoryManager>();
|
||||
|
||||
// Bot
|
||||
services.AddSingleton<FramePipelineService>();
|
||||
services.AddSingleton<LinkManager>();
|
||||
services.AddSingleton<TradeExecutor>();
|
||||
services.AddSingleton<TradeQueue>();
|
||||
|
|
|
|||
|
|
@ -1,25 +1,76 @@
|
|||
using Timer = System.Timers.Timer;
|
||||
using Avalonia.Threading;
|
||||
using CommunityToolkit.Mvvm.ComponentModel;
|
||||
using Poe2Trade.Bot;
|
||||
using Poe2Trade.Core;
|
||||
using Poe2Trade.Screen;
|
||||
|
||||
namespace Poe2Trade.Ui.ViewModels;
|
||||
|
||||
public partial class MappingViewModel : ObservableObject
|
||||
public partial class MappingViewModel : ObservableObject, IDisposable
|
||||
{
|
||||
private readonly BotOrchestrator _bot;
|
||||
private readonly Timer _statsTimer;
|
||||
|
||||
[ObservableProperty] private MapType _selectedMapType;
|
||||
[ObservableProperty] private bool _isFrameSaverEnabled;
|
||||
[ObservableProperty] private int _framesSaved;
|
||||
[ObservableProperty] private bool _isDetectionEnabled;
|
||||
[ObservableProperty] private int _enemiesDetected;
|
||||
[ObservableProperty] private float _inferenceMs;
|
||||
[ObservableProperty] private bool _hasModel;
|
||||
|
||||
public static MapType[] MapTypes { get; } = [MapType.TrialOfChaos, MapType.Temple, MapType.Endgame];
|
||||
|
||||
private static readonly string ModelPath = Path.GetFullPath("tools/python-detect/models/enemy-v1.pt");
|
||||
|
||||
public MappingViewModel(BotOrchestrator bot)
|
||||
{
|
||||
_bot = bot;
|
||||
_selectedMapType = bot.Config.MapType;
|
||||
_hasModel = File.Exists(ModelPath);
|
||||
|
||||
_bot.EnemyDetector.DetectionUpdated += OnDetectionUpdated;
|
||||
|
||||
_statsTimer = new Timer(1000);
|
||||
_statsTimer.Elapsed += (_, _) => Dispatcher.UIThread.Post(RefreshStats);
|
||||
_statsTimer.Start();
|
||||
}
|
||||
|
||||
partial void OnSelectedMapTypeChanged(MapType value)
|
||||
{
|
||||
_bot.Store.UpdateSettings(s => s.MapType = value);
|
||||
}
|
||||
|
||||
partial void OnIsFrameSaverEnabledChanged(bool value)
|
||||
{
|
||||
_bot.FrameSaver.Enabled = value;
|
||||
}
|
||||
|
||||
partial void OnIsDetectionEnabledChanged(bool value)
|
||||
{
|
||||
_bot.EnemyDetector.Enabled = value;
|
||||
}
|
||||
|
||||
private void OnDetectionUpdated(DetectionSnapshot snapshot)
|
||||
{
|
||||
Dispatcher.UIThread.Post(() =>
|
||||
{
|
||||
EnemiesDetected = snapshot.Enemies.Count;
|
||||
InferenceMs = snapshot.InferenceMs;
|
||||
});
|
||||
}
|
||||
|
||||
private void RefreshStats()
|
||||
{
|
||||
FramesSaved = _bot.FrameSaver.SavedCount;
|
||||
HasModel = File.Exists(ModelPath);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_statsTimer.Stop();
|
||||
_statsTimer.Dispose();
|
||||
_bot.EnemyDetector.DetectionUpdated -= OnDetectionUpdated;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -213,10 +213,13 @@
|
|||
|
||||
<!-- ========== MAPPING TAB ========== -->
|
||||
<TabItem Header="Mapping">
|
||||
<Border DataContext="{Binding MappingVm}" Background="#161b22"
|
||||
BorderBrush="#30363d" BorderThickness="1" CornerRadius="8"
|
||||
Padding="10" Margin="0,6,0,0">
|
||||
<StackPanel Spacing="8" x:DataType="vm:MappingViewModel">
|
||||
<ScrollViewer DataContext="{Binding MappingVm}" Margin="0,6,0,0">
|
||||
<StackPanel Spacing="8" Margin="0" x:DataType="vm:MappingViewModel">
|
||||
|
||||
<!-- Map Type -->
|
||||
<Border Background="#161b22" BorderBrush="#30363d" BorderThickness="1"
|
||||
CornerRadius="8" Padding="10">
|
||||
<StackPanel Spacing="8">
|
||||
<TextBlock Text="MAP TYPE" FontSize="11" FontWeight="SemiBold"
|
||||
Foreground="#8b949e" />
|
||||
<ComboBox ItemsSource="{x:Static vm:MappingViewModel.MapTypes}"
|
||||
|
|
@ -227,6 +230,49 @@
|
|||
FontSize="13" Foreground="#e6edf3" />
|
||||
</StackPanel>
|
||||
</Border>
|
||||
|
||||
<!-- Training Data -->
|
||||
<Border Background="#161b22" BorderBrush="#30363d" BorderThickness="1"
|
||||
CornerRadius="8" Padding="10">
|
||||
<StackPanel Spacing="8">
|
||||
<TextBlock Text="TRAINING DATA" FontSize="11" FontWeight="SemiBold"
|
||||
Foreground="#8b949e" />
|
||||
<ToggleSwitch IsChecked="{Binding IsFrameSaverEnabled}"
|
||||
OnContent="Save Frames" OffContent="Save Frames" />
|
||||
<TextBlock FontSize="12" Foreground="#8b949e">
|
||||
<TextBlock.Text>
|
||||
<MultiBinding StringFormat="{}{0} frames saved to training-data/raw/">
|
||||
<Binding Path="FramesSaved" />
|
||||
</MultiBinding>
|
||||
</TextBlock.Text>
|
||||
</TextBlock>
|
||||
</StackPanel>
|
||||
</Border>
|
||||
|
||||
<!-- Enemy Detection -->
|
||||
<Border Background="#161b22" BorderBrush="#30363d" BorderThickness="1"
|
||||
CornerRadius="8" Padding="10">
|
||||
<StackPanel Spacing="8">
|
||||
<TextBlock Text="ENEMY DETECTION" FontSize="11" FontWeight="SemiBold"
|
||||
Foreground="#8b949e" />
|
||||
<ToggleSwitch IsChecked="{Binding IsDetectionEnabled}"
|
||||
IsEnabled="{Binding HasModel}"
|
||||
OnContent="Detection On" OffContent="Detection Off" />
|
||||
<StackPanel Orientation="Horizontal" Spacing="16"
|
||||
IsVisible="{Binding HasModel}">
|
||||
<TextBlock Text="{Binding EnemiesDetected, StringFormat='{}{0} enemies'}"
|
||||
FontSize="12" Foreground="#e6edf3" />
|
||||
<TextBlock Text="{Binding InferenceMs, StringFormat='{}{0:F1}ms inference'}"
|
||||
FontSize="12" Foreground="#8b949e" />
|
||||
</StackPanel>
|
||||
<TextBlock Text="No model found. Train one first."
|
||||
FontSize="12" Foreground="#484f58"
|
||||
IsVisible="{Binding !HasModel}" />
|
||||
</StackPanel>
|
||||
</Border>
|
||||
|
||||
</StackPanel>
|
||||
</ScrollViewer>
|
||||
</TabItem>
|
||||
|
||||
<!-- ========== DEBUG TAB ========== -->
|
||||
|
|
|
|||
147
tools/python-detect/daemon.py
Normal file
147
tools/python-detect/daemon.py
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
"""
|
||||
Persistent Python YOLO detection daemon (stdin/stdout JSON-per-line protocol).
|
||||
|
||||
Loads a YOLOv11 model and serves inference requests over stdin/stdout.
|
||||
Managed as a subprocess by PythonDetectBridge in Poe2Trade.Screen.
|
||||
|
||||
Request: {"cmd": "detect", "imageBase64": "...", "conf": 0.3, "iou": 0.45, "imgsz": 640}
|
||||
Response: {"ok": true, "count": 3, "inferenceMs": 12.5, "detections": [...]}
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
import time
|
||||
|
||||
_model = None
|
||||
|
||||
|
||||
def _redirect_stdout_to_stderr():
|
||||
"""Redirect stdout to stderr so library print() calls don't corrupt the JSON protocol."""
|
||||
real_stdout = sys.stdout
|
||||
sys.stdout = sys.stderr
|
||||
return real_stdout
|
||||
|
||||
|
||||
def _restore_stdout(real_stdout):
|
||||
sys.stdout = real_stdout
|
||||
|
||||
|
||||
def load_model():
|
||||
global _model
|
||||
if _model is not None:
|
||||
return _model
|
||||
|
||||
import os
|
||||
from ultralytics import YOLO
|
||||
|
||||
model_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "models")
|
||||
model_path = os.path.join(model_dir, "enemy-v1.pt")
|
||||
|
||||
if not os.path.exists(model_path):
|
||||
raise FileNotFoundError(f"Model not found: {model_path}")
|
||||
|
||||
sys.stderr.write(f"Loading YOLO model from {model_path}...\n")
|
||||
sys.stderr.flush()
|
||||
|
||||
real_stdout = _redirect_stdout_to_stderr()
|
||||
try:
|
||||
_model = YOLO(model_path)
|
||||
# Warmup with dummy inference (triggers CUDA init)
|
||||
import numpy as np
|
||||
dummy = np.zeros((640, 640, 3), dtype=np.uint8)
|
||||
_model.predict(dummy, verbose=False)
|
||||
finally:
|
||||
_restore_stdout(real_stdout)
|
||||
|
||||
sys.stderr.write("YOLO model loaded and warmed up.\n")
|
||||
sys.stderr.flush()
|
||||
return _model
|
||||
|
||||
|
||||
def handle_detect(req):
|
||||
import base64
|
||||
import io
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
image_base64 = req.get("imageBase64")
|
||||
if not image_base64:
|
||||
return {"ok": False, "error": "Missing imageBase64"}
|
||||
|
||||
img_bytes = base64.b64decode(image_base64)
|
||||
img = np.array(Image.open(io.BytesIO(img_bytes)))
|
||||
|
||||
conf = req.get("conf", 0.3)
|
||||
iou = req.get("iou", 0.45)
|
||||
imgsz = req.get("imgsz", 640)
|
||||
|
||||
model = load_model()
|
||||
|
||||
real_stdout = _redirect_stdout_to_stderr()
|
||||
try:
|
||||
start = time.perf_counter()
|
||||
results = model.predict(img, conf=conf, iou=iou, imgsz=imgsz, verbose=False)
|
||||
inference_ms = (time.perf_counter() - start) * 1000
|
||||
finally:
|
||||
_restore_stdout(real_stdout)
|
||||
|
||||
detections = []
|
||||
for result in results:
|
||||
boxes = result.boxes
|
||||
if boxes is None:
|
||||
continue
|
||||
for i in range(len(boxes)):
|
||||
box = boxes[i]
|
||||
x1, y1, x2, y2 = box.xyxy[0].tolist()
|
||||
x, y = int(x1), int(y1)
|
||||
w, h = int(x2 - x1), int(y2 - y1)
|
||||
cx, cy = x + w // 2, y + h // 2
|
||||
class_id = int(box.cls[0].item())
|
||||
class_name = result.names[class_id] if result.names else str(class_id)
|
||||
confidence = float(box.conf[0].item())
|
||||
|
||||
detections.append({
|
||||
"class": class_name,
|
||||
"classId": class_id,
|
||||
"confidence": round(confidence, 4),
|
||||
"x": x, "y": y, "width": w, "height": h,
|
||||
"cx": cx, "cy": cy,
|
||||
})
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"count": len(detections),
|
||||
"inferenceMs": round(inference_ms, 2),
|
||||
"detections": detections,
|
||||
}
|
||||
|
||||
|
||||
def handle_request(req):
|
||||
cmd = req.get("cmd")
|
||||
if cmd == "detect":
|
||||
return handle_detect(req)
|
||||
if cmd == "ping":
|
||||
return {"ok": True, "pong": True}
|
||||
return {"ok": False, "error": f"Unknown command: {cmd}"}
|
||||
|
||||
|
||||
def main():
|
||||
# Signal ready immediately — model loads lazily on first detect request
|
||||
sys.stdout.write(json.dumps({"ok": True, "ready": True}) + "\n")
|
||||
sys.stdout.flush()
|
||||
|
||||
for line in sys.stdin:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
req = json.loads(line)
|
||||
resp = handle_request(req)
|
||||
except Exception as e:
|
||||
resp = {"ok": False, "error": str(e)}
|
||||
sys.stdout.write(json.dumps(resp) + "\n")
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
5
tools/python-detect/requirements.txt
Normal file
5
tools/python-detect/requirements.txt
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
ultralytics
|
||||
torch
|
||||
torchvision
|
||||
numpy
|
||||
pillow
|
||||
20
tools/python-detect/setup-venv.bat
Normal file
20
tools/python-detect/setup-venv.bat
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
@echo off
|
||||
setlocal
|
||||
|
||||
cd /d "%~dp0"
|
||||
|
||||
if exist .venv (
|
||||
echo Existing venv found, upgrading pip...
|
||||
.venv\Scripts\python.exe -m pip install --upgrade pip
|
||||
) else (
|
||||
echo Creating virtual environment...
|
||||
python -m venv .venv
|
||||
.venv\Scripts\python.exe -m pip install --upgrade pip
|
||||
)
|
||||
|
||||
echo Installing dependencies...
|
||||
.venv\Scripts\pip.exe install -r requirements.txt
|
||||
|
||||
echo.
|
||||
echo Setup complete. Activate with: .venv\Scripts\activate
|
||||
echo Test daemon with: echo {"cmd":"ping"} | .venv\Scripts\python.exe daemon.py
|
||||
58
tools/python-detect/train.py
Normal file
58
tools/python-detect/train.py
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
"""
|
||||
Training script for YOLOv11n enemy detection model.
|
||||
|
||||
Usage:
|
||||
python train.py --data path/to/data.yaml --epochs 100
|
||||
|
||||
Expects YOLO-format dataset with data.yaml pointing to train/val image directories.
|
||||
Export from Roboflow in "YOLOv11" format.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Train YOLOv11n enemy detector")
|
||||
parser.add_argument("--data", required=True, help="Path to data.yaml")
|
||||
parser.add_argument("--epochs", type=int, default=100, help="Training epochs")
|
||||
parser.add_argument("--imgsz", type=int, default=640, help="Image size")
|
||||
parser.add_argument("--batch", type=int, default=16, help="Batch size")
|
||||
parser.add_argument("--device", default="0", help="CUDA device (0, cpu)")
|
||||
parser.add_argument("--name", default="enemy-v1", help="Run name")
|
||||
args = parser.parse_args()
|
||||
|
||||
from ultralytics import YOLO
|
||||
|
||||
model = YOLO("yolo11n.pt") # start from pretrained nano
|
||||
|
||||
model.train(
|
||||
data=args.data,
|
||||
epochs=args.epochs,
|
||||
imgsz=args.imgsz,
|
||||
batch=args.batch,
|
||||
device=args.device,
|
||||
name=args.name,
|
||||
patience=20, # early stopping
|
||||
save=True,
|
||||
save_period=10,
|
||||
plots=True,
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
# Copy best weights to models directory
|
||||
best_path = os.path.join("runs", "detect", args.name, "weights", "best.pt")
|
||||
output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "models")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
output_path = os.path.join(output_dir, f"{args.name}.pt")
|
||||
if os.path.exists(best_path):
|
||||
import shutil
|
||||
shutil.copy2(best_path, output_path)
|
||||
print(f"\nBest model copied to: {output_path}")
|
||||
else:
|
||||
print(f"\nWarning: {best_path} not found — check training output")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Loading…
Add table
Add a link
Reference in a new issue