simulation done

docs/data-and-memory.md

@@ -0,0 +1,229 @@
+# Nexus.Data & Nexus.Memory — The Data Pipeline
+
+## Overview
+
+Two-layer architecture: **Memory** reads raw bytes from the game process; **Data** interprets, classifies, and caches them. This separation keeps memory reading logic free of business rules and makes the data layer testable independently.
+
+```
+Game Process (RPM)
+    │
+    ▼
+Nexus.Memory (raw reads, no business logic)
+    │  GameMemoryReader → hierarchical state tree
+    │  EntityList → red-black tree traversal
+    │  ComponentReader → ECS component extraction
+    │
+    ▼
+Nexus.Data (interpretation, classification, caching)
+    │  MemoryPoller → two-tier event loop
+    │  EntityMapper → Memory.Entity → Core.EntitySnapshot
+    │  EntityClassifier → path + components → EntityCategory
+    │  GameStateEnricher → derived threat/danger metrics
+    │
+    ▼
+GameDataCache (volatile references, lock-free)
+    │
+    ▼
+Bot Systems (read-only consumers)
+```
+
+---
+
+## GameDataCache — Single Source of Truth
+
+Thread-safe, volatile reference holder. Writer: MemoryPoller thread. Readers: bot systems.
+
+**Hot fields** (updated at 60Hz — 4 lightweight RPM calls):
+- CameraMatrix (64 bytes)
+- PlayerPosition (X, Y, Z)
+- PlayerVitals (HP, mana, ES current/max)
+- IsLoading, IsEscapeOpen
+
+**Cold fields** (updated at 10Hz — full hierarchical read):
+- Entities, HostileMonsters, NearbyLoot
+- Terrain (WalkabilitySnapshot)
+- AreaHash, AreaLevel, CurrentAreaName
+- Quest data (linked lists, UI groups, state entries)
+- LatestState (complete GameState)
+
+**Slow fields** (updated at 1Hz):
+- Character name
+- Quest linked lists, quest state entries
+
+No locks — relies on volatile reference semantics for atomic swaps.
+
+---
+
+## MemoryPoller — Two-Tier Event Loop
+
+Owns the memory-reading background thread.
+
+### Hot Tick (60Hz)
+
+4 pre-resolved RPM calls using cached addresses:
+1. Camera matrix (64 bytes from cached address)
+2. Player position (12 bytes: X, Y, Z)
+3. Player vitals (24 bytes: HP, mana, ES)
+4. Loading/escape state (pointer dereference + int)
+
+No allocations, no GC. Targeting ~3-5ms per tick.
+
+### Cold Tick (10Hz, every 6th hot tick)
+
+Full hierarchical read:
+1. `GameMemoryReader.ReadSnapshot()` — cascades through state tree
+2. Re-resolve hot addresses via `ResolveHotAddresses()`
+3. `BuildGameState()` — map entities, filter lists, process quests
+4. `GameStateEnricher.Enrich()` — compute derived fields
+5. Update all cache fields
+
+### BuildGameState() Flow
+
+```
+ReadSnapshot() → GameStateSnapshot (raw)
+    │
+    ├── Entity mapping:
+    │   for each entity in snapshot:
+    │     EntityMapper.MapEntity(entity, playerPos) → EntitySnapshot
+    │
+    ├── Filter into:
+    │   - HostileMonsters (Category==Monster && IsAlive)
+    │   - NearbyLoot (Category==WorldItem)
+    │   - All entities
+    │
+    ├── Quest processing:
+    │   - Filter active (StateId > 0)
+    │   - Resolve state text via QuestStateLookup
+    │   - Convert to QuestProgress, QuestInfo
+    │
+    └── Returns GameState
+```
+
+---
+
+## GameMemoryReader — Hierarchical State Tree
+
+Top-level orchestrator. Creates sub-readers on `Attach()`:
+
+```
+GameStates (top)
+└── InGameState
+    ├── AreaInstance
+    │   ├── EntityList (MSVC std::map red-black tree)
+    │   ├── PlayerSkills (Actor component)
+    │   ├── QuestStates (dat file entries)
+    │   └── Terrain (walkability grid)
+    ├── UIElements (quest linked lists, UI tree)
+    └── WorldData (camera matrix)
+```
+
+Each `RemoteObject` caches its data and depends on parent for context. Single `Update()` call cascades through tree.
+
+### Infrastructure
+
+- **ProcessMemory**: P/Invoke wrapper for ReadProcessMemory. Tracks reads/sec and KB/sec.
+- **MemoryContext**: Shared state — process handle, offsets, module base, pattern scanner.
+- **ComponentReader**: Reads ECS components (Life, Render, Mods, etc.) from entities.
+- **MsvcStringReader**: Reads MSVC std::wstring (SSO-aware: inline if capacity ≤ 8, heap pointer otherwise).
+- **PatternScanner**: AOB scan for resolving base addresses.
+
+---
+
+## EntityList — Tree Traversal
+
+Reads entities from AreaInstance's MSVC std::map (red-black tree, in-order traversal).
+
+**Tree node layout:**
+```
++0x00: left child ptr
++0x08: parent ptr
++0x10: right child ptr
++0x28: entity pointer
+```
+
+**Optimization**: Tree order is cached — re-walked only when entity count changes.
+
+**Per-entity reads:**
+1. Path (EntityDetails → std::wstring)
+2. Skip low-priority types (effects, terrain, critters — no components read)
+3. Position (Render component: X, Y, Z)
+4. Component lookup (STL hash map: name → index)
+5. Component data:
+   - Targetable (bool flag)
+   - Mods/ObjectMagicProperties (rarity)
+   - Life (HP, dynamic — re-read every frame for monsters)
+   - Actor (action ID)
+   - WorldItem (inner entity for ground loot)
+   - AreaTransition (destination area)
+
+**Caching strategy:**
+- Stable per entity: path, component list, targetable, rarity, transition name
+- Dynamic (re-read every frame): monster HP, action ID
+
+---
+
+## EntityClassifier — Path + Components → Category
+
+Single source of truth for entity classification.
+
+1. **Path-based** (primary): Parses `Metadata/[Category]/...` path segments
+2. **Component override**: Monster, Chest, Shrine, Waypoint, AreaTransition, Portal, TownPortal, NPC, Player
+
+Output: `EntityCategory` (Core enum, 17 types)
+
+---
+
+## EntityMapper — Memory.Entity → Core.EntitySnapshot
+
+Transforms raw memory data to enriched snapshots:
+
+```
+Memory.Entity (raw)
+    │
+    ├── Copy: ID, path, metadata, position, Z, vitals, components, mods
+    ├── Classify: EntityClassifier.Classify(path, components) → EntityCategory
+    ├── Threat level: Rarity → MonsterThreatLevel
+    ├── Area name: AreaNameLookup.Resolve() for transitions
+    ├── Distance: Vector2.Distance(position, playerPos)
+    └── Alive state: HasVitals ? LifeCurrent > 0 : true
+    │
+    ▼
+Core.EntitySnapshot (public, classified, enriched)
+```
+
+---
+
+## GameStateEnricher — Derived Metrics
+
+Computed once per cold tick, before systems run.
+
+**NearestEnemies**: HostileMonsters sorted by distance to player.
+
+**ThreatMap**:
+- TotalHostiles, CloseRange (<300u), MidRange (300-600u), FarRange (600-1200u)
+- ClosestDistance, ThreatCentroid (position average), HasRareOrUnique
+
+**DangerLevel** — Weighted threat score:
+```
+score = Σ (distance_weight × rarity_multiplier)
+
+Distance weights:  <200u = 3×,  <400u = 2×,  else = 1×
+Rarity multipliers: Unique=5×, Rare=3×, Magic=1.5×, White=1×
+
+Life override:  HP < 30% → Critical,  HP < 50% → High
+Score thresholds: ≥15 = Critical, ≥8 = High, ≥4 = Medium, >0 = Low, 0 = Safe
+```
+
+---
+
+## Key Architectural Patterns
+
+| Pattern | Implementation |
+|---------|---------------|
+| Lock-free cross-thread | Volatile references in GameDataCache; no locks needed |
+| Two-tier polling | Hot (4 RPM, 60Hz) + Cold (full read, 10Hz) |
+| Hierarchical caching | Each RemoteObject caches data, re-reads only on change |
+| Entity caching | Stable data cached per entity/zone; dynamic data (HP) re-read per frame |
+| Separation of concerns | Memory: raw bytes. Data: interpretation + classification |
+| Area name resolution | AreaNameLookup loads areas.json, caches ID → display name |
+| Area graph | BFS pathfinding for quest progression ordering |