work on qm spider

apps/stock/data-ingestion/test-spider.md

@@ -0,0 +1,80 @@
+# QM Spider Symbol Search Test
+
+## How the Spider Search Works
+
+The spider search is a recursive symbol discovery system that explores the QM API systematically:
+
+### 1. Root Job (Weekly Schedule)
+```typescript
+// Triggered every Saturday at midnight
+@ScheduledOperation('spider-symbol-search', '0 0 * * 6', {...})
+
+// Creates 26 jobs: A, B, C, ... Z
+input: { prefix: null, depth: 0, maxDepth: 4 }
+```
+
+### 2. Level 1 - Single Letters (A-Z)
+```typescript
+// Each job searches for symbols starting with that letter
+input: { prefix: 'A', depth: 1, maxDepth: 4 }
+
+// If 10+ symbols found, creates: AA, AB, AC, ... AZ
+```
+
+### 3. Level 2 - Two Letters (AA-ZZ)
+```typescript
+// Searches for symbols starting with two letters
+input: { prefix: 'AA', depth: 2, maxDepth: 4 }
+
+// If 10+ symbols found, creates: AAA, AAB, AAC, ... AAZ
+```
+
+### 4. Level 3 & 4 - Deeper Search
+Continues until `maxDepth` is reached or fewer than 10 symbols are found.
+
+## Testing the Spider
+
+### Manual Test - Root Job
+```bash
+# Trigger spider search root job
+curl -X POST http://localhost:3000/api/handlers/qm/operations/spider-symbol-search \
+  -H "Content-Type: application/json" \
+  -d '{}'
+```
+
+### Manual Test - Specific Prefix
+```bash
+# Search for symbols starting with "APP"
+curl -X POST http://localhost:3000/api/handlers/qm/operations/spider-symbol-search \
+  -H "Content-Type: application/json" \
+  -d '{"prefix": "APP", "depth": 3, "maxDepth": 4}'
+```
+
+### Direct Symbol Search
+```bash
+# Search for specific symbols
+curl -X POST http://localhost:3000/api/handlers/qm/operations/search-symbols \
+  -H "Content-Type: application/json" \
+  -d '{"query": "AAPL"}'
+```
+
+## Optimization Features
+
+1. **Intelligent Branching**: Only creates child jobs if 10+ symbols found
+2. **Priority Management**: Deeper searches get lower priority
+3. **Staggered Execution**: Jobs are delayed to avoid API rate limits
+4. **Session Management**: Uses QM sessions with failure tracking
+5. **Caching**: Results cached for 30 minutes to avoid duplicate API calls
+
+## MongoDB Collections
+
+- `qm_symbols`: Stores discovered symbols with metadata
+- `qm_exchanges`: Stores unique exchanges found during searches
+
+## Monitoring
+
+Check logs for:
+- "Spider symbol search" - Job execution
+- "Created root spider jobs" - Initial A-Z creation
+- "Stored symbols from spider search" - Successful symbol storage
+- "Found X symbols for Y, queued Z child jobs" - Branching decisions