229 lines
7.3 KiB
Markdown
229 lines
7.3 KiB
Markdown
# Nexus.Data & Nexus.Memory — The Data Pipeline
|
||
|
||
## Overview
|
||
|
||
Two-layer architecture: **Memory** reads raw bytes from the game process; **Data** interprets, classifies, and caches them. This separation keeps memory reading logic free of business rules and makes the data layer testable independently.
|
||
|
||
```
|
||
Game Process (RPM)
|
||
│
|
||
▼
|
||
Nexus.Memory (raw reads, no business logic)
|
||
│ GameMemoryReader → hierarchical state tree
|
||
│ EntityList → red-black tree traversal
|
||
│ ComponentReader → ECS component extraction
|
||
│
|
||
▼
|
||
Nexus.Data (interpretation, classification, caching)
|
||
│ MemoryPoller → two-tier event loop
|
||
│ EntityMapper → Memory.Entity → Core.EntitySnapshot
|
||
│ EntityClassifier → path + components → EntityCategory
|
||
│ GameStateEnricher → derived threat/danger metrics
|
||
│
|
||
▼
|
||
GameDataCache (volatile references, lock-free)
|
||
│
|
||
▼
|
||
Bot Systems (read-only consumers)
|
||
```
|
||
|
||
---
|
||
|
||
## GameDataCache — Single Source of Truth
|
||
|
||
Thread-safe, volatile reference holder. Writer: MemoryPoller thread. Readers: bot systems.
|
||
|
||
**Hot fields** (updated at 60Hz — 4 lightweight RPM calls):
|
||
- CameraMatrix (64 bytes)
|
||
- PlayerPosition (X, Y, Z)
|
||
- PlayerVitals (HP, mana, ES current/max)
|
||
- IsLoading, IsEscapeOpen
|
||
|
||
**Cold fields** (updated at 10Hz — full hierarchical read):
|
||
- Entities, HostileMonsters, NearbyLoot
|
||
- Terrain (WalkabilitySnapshot)
|
||
- AreaHash, AreaLevel, CurrentAreaName
|
||
- Quest data (linked lists, UI groups, state entries)
|
||
- LatestState (complete GameState)
|
||
|
||
**Slow fields** (updated at 1Hz):
|
||
- Character name
|
||
- Quest linked lists, quest state entries
|
||
|
||
No locks — relies on volatile reference semantics for atomic swaps.
|
||
|
||
---
|
||
|
||
## MemoryPoller — Two-Tier Event Loop
|
||
|
||
Owns the memory-reading background thread.
|
||
|
||
### Hot Tick (60Hz)
|
||
|
||
4 pre-resolved RPM calls using cached addresses:
|
||
1. Camera matrix (64 bytes from cached address)
|
||
2. Player position (12 bytes: X, Y, Z)
|
||
3. Player vitals (24 bytes: HP, mana, ES)
|
||
4. Loading/escape state (pointer dereference + int)
|
||
|
||
No allocations, no GC. Targeting ~3-5ms per tick.
|
||
|
||
### Cold Tick (10Hz, every 6th hot tick)
|
||
|
||
Full hierarchical read:
|
||
1. `GameMemoryReader.ReadSnapshot()` — cascades through state tree
|
||
2. Re-resolve hot addresses via `ResolveHotAddresses()`
|
||
3. `BuildGameState()` — map entities, filter lists, process quests
|
||
4. `GameStateEnricher.Enrich()` — compute derived fields
|
||
5. Update all cache fields
|
||
|
||
### BuildGameState() Flow
|
||
|
||
```
|
||
ReadSnapshot() → GameStateSnapshot (raw)
|
||
│
|
||
├── Entity mapping:
|
||
│ for each entity in snapshot:
|
||
│ EntityMapper.MapEntity(entity, playerPos) → EntitySnapshot
|
||
│
|
||
├── Filter into:
|
||
│ - HostileMonsters (Category==Monster && IsAlive)
|
||
│ - NearbyLoot (Category==WorldItem)
|
||
│ - All entities
|
||
│
|
||
├── Quest processing:
|
||
│ - Filter active (StateId > 0)
|
||
│ - Resolve state text via QuestStateLookup
|
||
│ - Convert to QuestProgress, QuestInfo
|
||
│
|
||
└── Returns GameState
|
||
```
|
||
|
||
---
|
||
|
||
## GameMemoryReader — Hierarchical State Tree
|
||
|
||
Top-level orchestrator. Creates sub-readers on `Attach()`:
|
||
|
||
```
|
||
GameStates (top)
|
||
└── InGameState
|
||
├── AreaInstance
|
||
│ ├── EntityList (MSVC std::map red-black tree)
|
||
│ ├── PlayerSkills (Actor component)
|
||
│ ├── QuestStates (dat file entries)
|
||
│ └── Terrain (walkability grid)
|
||
├── UIElements (quest linked lists, UI tree)
|
||
└── WorldData (camera matrix)
|
||
```
|
||
|
||
Each `RemoteObject` caches its data and depends on parent for context. Single `Update()` call cascades through tree.
|
||
|
||
### Infrastructure
|
||
|
||
- **ProcessMemory**: P/Invoke wrapper for ReadProcessMemory. Tracks reads/sec and KB/sec.
|
||
- **MemoryContext**: Shared state — process handle, offsets, module base, pattern scanner.
|
||
- **ComponentReader**: Reads ECS components (Life, Render, Mods, etc.) from entities.
|
||
- **MsvcStringReader**: Reads MSVC std::wstring (SSO-aware: inline if capacity ≤ 8, heap pointer otherwise).
|
||
- **PatternScanner**: AOB scan for resolving base addresses.
|
||
|
||
---
|
||
|
||
## EntityList — Tree Traversal
|
||
|
||
Reads entities from AreaInstance's MSVC std::map (red-black tree, in-order traversal).
|
||
|
||
**Tree node layout:**
|
||
```
|
||
+0x00: left child ptr
|
||
+0x08: parent ptr
|
||
+0x10: right child ptr
|
||
+0x28: entity pointer
|
||
```
|
||
|
||
**Optimization**: Tree order is cached — re-walked only when entity count changes.
|
||
|
||
**Per-entity reads:**
|
||
1. Path (EntityDetails → std::wstring)
|
||
2. Skip low-priority types (effects, terrain, critters — no components read)
|
||
3. Position (Render component: X, Y, Z)
|
||
4. Component lookup (STL hash map: name → index)
|
||
5. Component data:
|
||
- Targetable (bool flag)
|
||
- Mods/ObjectMagicProperties (rarity)
|
||
- Life (HP, dynamic — re-read every frame for monsters)
|
||
- Actor (action ID)
|
||
- WorldItem (inner entity for ground loot)
|
||
- AreaTransition (destination area)
|
||
|
||
**Caching strategy:**
|
||
- Stable per entity: path, component list, targetable, rarity, transition name
|
||
- Dynamic (re-read every frame): monster HP, action ID
|
||
|
||
---
|
||
|
||
## EntityClassifier — Path + Components → Category
|
||
|
||
Single source of truth for entity classification.
|
||
|
||
1. **Path-based** (primary): Parses `Metadata/[Category]/...` path segments
|
||
2. **Component override**: Monster, Chest, Shrine, Waypoint, AreaTransition, Portal, TownPortal, NPC, Player
|
||
|
||
Output: `EntityCategory` (Core enum, 17 types)
|
||
|
||
---
|
||
|
||
## EntityMapper — Memory.Entity → Core.EntitySnapshot
|
||
|
||
Transforms raw memory data to enriched snapshots:
|
||
|
||
```
|
||
Memory.Entity (raw)
|
||
│
|
||
├── Copy: ID, path, metadata, position, Z, vitals, components, mods
|
||
├── Classify: EntityClassifier.Classify(path, components) → EntityCategory
|
||
├── Threat level: Rarity → MonsterThreatLevel
|
||
├── Area name: AreaNameLookup.Resolve() for transitions
|
||
├── Distance: Vector2.Distance(position, playerPos)
|
||
└── Alive state: HasVitals ? LifeCurrent > 0 : true
|
||
│
|
||
▼
|
||
Core.EntitySnapshot (public, classified, enriched)
|
||
```
|
||
|
||
---
|
||
|
||
## GameStateEnricher — Derived Metrics
|
||
|
||
Computed once per cold tick, before systems run.
|
||
|
||
**NearestEnemies**: HostileMonsters sorted by distance to player.
|
||
|
||
**ThreatMap**:
|
||
- TotalHostiles, CloseRange (<300u), MidRange (300-600u), FarRange (600-1200u)
|
||
- ClosestDistance, ThreatCentroid (position average), HasRareOrUnique
|
||
|
||
**DangerLevel** — Weighted threat score:
|
||
```
|
||
score = Σ (distance_weight × rarity_multiplier)
|
||
|
||
Distance weights: <200u = 3×, <400u = 2×, else = 1×
|
||
Rarity multipliers: Unique=5×, Rare=3×, Magic=1.5×, White=1×
|
||
|
||
Life override: HP < 30% → Critical, HP < 50% → High
|
||
Score thresholds: ≥15 = Critical, ≥8 = High, ≥4 = Medium, >0 = Low, 0 = Safe
|
||
```
|
||
|
||
---
|
||
|
||
## Key Architectural Patterns
|
||
|
||
| Pattern | Implementation |
|
||
|---------|---------------|
|
||
| Lock-free cross-thread | Volatile references in GameDataCache; no locks needed |
|
||
| Two-tier polling | Hot (4 RPM, 60Hz) + Cold (full read, 10Hz) |
|
||
| Hierarchical caching | Each RemoteObject caches data, re-reads only on change |
|
||
| Entity caching | Stable data cached per entity/zone; dynamic data (HP) re-read per frame |
|
||
| Separation of concerns | Memory: raw bytes. Data: interpretation + classification |
|
||
| Area name resolution | AreaNameLookup loads areas.json, caches ID → display name |
|
||
| Area graph | BFS pathfinding for quest progression ordering |
|