poe2-bot/docs/data-and-memory.md

Nexus.Data & Nexus.Memory — The Data Pipeline

Overview

Two-layer architecture: Memory reads raw bytes from the game process; Data interprets, classifies, and caches them. This separation keeps memory reading logic free of business rules and makes the data layer testable independently.

Game Process (RPM)
    │
    ▼
Nexus.Memory (raw reads, no business logic)
    │  GameMemoryReader → hierarchical state tree
    │  EntityList → red-black tree traversal
    │  ComponentReader → ECS component extraction
    │
    ▼
Nexus.Data (interpretation, classification, caching)
    │  MemoryPoller → two-tier event loop
    │  EntityMapper → Memory.Entity → Core.EntitySnapshot
    │  EntityClassifier → path + components → EntityCategory
    │  GameStateEnricher → derived threat/danger metrics
    │
    ▼
GameDataCache (volatile references, lock-free)
    │
    ▼
Bot Systems (read-only consumers)

---

GameDataCache — Single Source of Truth

Thread-safe, volatile reference holder. Writer: MemoryPoller thread. Readers: bot systems.

Hot fields (updated at 60Hz — 4 lightweight RPM calls):

• CameraMatrix (64 bytes)
• PlayerPosition (X, Y, Z)
• PlayerVitals (HP, mana, ES current/max)
• IsLoading, IsEscapeOpen

Cold fields (updated at 10Hz — full hierarchical read):

• Entities, HostileMonsters, NearbyLoot
• Terrain (WalkabilitySnapshot)
• AreaHash, AreaLevel, CurrentAreaName
• Quest data (linked lists, UI groups, state entries)
• LatestState (complete GameState)

Slow fields (updated at 1Hz):

• Character name
• Quest linked lists, quest state entries

No locks — relies on volatile reference semantics for atomic swaps.

---

MemoryPoller — Two-Tier Event Loop

Owns the memory-reading background thread.

Hot Tick (60Hz)

4 pre-resolved RPM calls using cached addresses:

1. Camera matrix (64 bytes from cached address)
2. Player position (12 bytes: X, Y, Z)
3. Player vitals (24 bytes: HP, mana, ES)
4. Loading/escape state (pointer dereference + int)

No allocations, no GC. Targeting ~3-5ms per tick.

Cold Tick (10Hz, every 6th hot tick)

Full hierarchical read:

1. GameMemoryReader.ReadSnapshot() — cascades through state tree
2. Re-resolve hot addresses via ResolveHotAddresses()
3. BuildGameState() — map entities, filter lists, process quests
4. GameStateEnricher.Enrich() — compute derived fields
5. Update all cache fields

BuildGameState() Flow

ReadSnapshot() → GameStateSnapshot (raw)
    │
    ├── Entity mapping:
    │   for each entity in snapshot:
    │     EntityMapper.MapEntity(entity, playerPos) → EntitySnapshot
    │
    ├── Filter into:
    │   - HostileMonsters (Category==Monster && IsAlive)
    │   - NearbyLoot (Category==WorldItem)
    │   - All entities
    │
    ├── Quest processing:
    │   - Filter active (StateId > 0)
    │   - Resolve state text via QuestStateLookup
    │   - Convert to QuestProgress, QuestInfo
    │
    └── Returns GameState

---

GameMemoryReader — Hierarchical State Tree

Top-level orchestrator. Creates sub-readers on Attach():

GameStates (top)
└── InGameState
    ├── AreaInstance
    │   ├── EntityList (MSVC std::map red-black tree)
    │   ├── PlayerSkills (Actor component)
    │   ├── QuestStates (dat file entries)
    │   └── Terrain (walkability grid)
    ├── UIElements (quest linked lists, UI tree)
    └── WorldData (camera matrix)

Each RemoteObject caches its data and depends on parent for context. Single Update() call cascades through tree.

Infrastructure

• ProcessMemory: P/Invoke wrapper for ReadProcessMemory. Tracks reads/sec and KB/sec.
• MemoryContext: Shared state — process handle, offsets, module base, pattern scanner.
• ComponentReader: Reads ECS components (Life, Render, Mods, etc.) from entities.
• MsvcStringReader: Reads MSVC std::wstring (SSO-aware: inline if capacity ≤ 8, heap pointer otherwise).
• PatternScanner: AOB scan for resolving base addresses.

---

EntityList — Tree Traversal

Reads entities from AreaInstance's MSVC std::map (red-black tree, in-order traversal).

Tree node layout:

+0x00: left child ptr
+0x08: parent ptr
+0x10: right child ptr
+0x28: entity pointer

Optimization: Tree order is cached — re-walked only when entity count changes.

Per-entity reads:

1. Path (EntityDetails → std::wstring)
2. Skip low-priority types (effects, terrain, critters — no components read)
3. Position (Render component: X, Y, Z)
4. Component lookup (STL hash map: name → index)
5. Component data:
   • Targetable (bool flag)
   • Mods/ObjectMagicProperties (rarity)
   • Life (HP, dynamic — re-read every frame for monsters)
   • Actor (action ID)
   • WorldItem (inner entity for ground loot)
   • AreaTransition (destination area)

Caching strategy:

• Stable per entity: path, component list, targetable, rarity, transition name
• Dynamic (re-read every frame): monster HP, action ID

---

EntityClassifier — Path + Components → Category

Single source of truth for entity classification.

1. Path-based (primary): Parses Metadata/[Category]/... path segments
2. Component override: Monster, Chest, Shrine, Waypoint, AreaTransition, Portal, TownPortal, NPC, Player

Output: EntityCategory (Core enum, 17 types)

---

EntityMapper — Memory.Entity → Core.EntitySnapshot

Transforms raw memory data to enriched snapshots:

Memory.Entity (raw)
    │
    ├── Copy: ID, path, metadata, position, Z, vitals, components, mods
    ├── Classify: EntityClassifier.Classify(path, components) → EntityCategory
    ├── Threat level: Rarity → MonsterThreatLevel
    ├── Area name: AreaNameLookup.Resolve() for transitions
    ├── Distance: Vector2.Distance(position, playerPos)
    └── Alive state: HasVitals ? LifeCurrent > 0 : true
    │
    ▼
Core.EntitySnapshot (public, classified, enriched)

---

GameStateEnricher — Derived Metrics

Computed once per cold tick, before systems run.

NearestEnemies: HostileMonsters sorted by distance to player.

ThreatMap:

• TotalHostiles, CloseRange (<300u), MidRange (300-600u), FarRange (600-1200u)
• ClosestDistance, ThreatCentroid (position average), HasRareOrUnique

DangerLevel — Weighted threat score:

score = Σ (distance_weight × rarity_multiplier)

Distance weights:  <200u = 3×,  <400u = 2×,  else = 1×
Rarity multipliers: Unique=5×, Rare=3×, Magic=1.5×, White=1×

Life override:  HP < 30% → Critical,  HP < 50% → High
Score thresholds: ≥15 = Critical, ≥8 = High, ≥4 = Medium, >0 = Low, 0 = Safe

---

Key Architectural Patterns

Pattern	Implementation
Lock-free cross-thread	Volatile references in GameDataCache; no locks needed
Two-tier polling	Hot (4 RPM, 60Hz) + Cold (full read, 10Hz)
Hierarchical caching	Each RemoteObject caches data, re-reads only on change
Entity caching	Stable data cached per entity/zone; dynamic data (HP) re-read per frame
Separation of concerns	Memory: raw bytes. Data: interpretation + classification
Area name resolution	AreaNameLookup loads areas.json, caches ID → display name
Area graph	BFS pathfinding for quest progression ordering