stock-bot/apps/stock/data-ingestion/test-spider.md

QM Spider Symbol Search Test

How the Spider Search Works

The spider search is a recursive symbol discovery system that explores the QM API systematically:

1. Root Job (Weekly Schedule)

// Triggered every Saturday at midnight
@ScheduledOperation('spider-symbol-search', '0 0 * * 6', {...})

// Creates 26 jobs: A, B, C, ... Z
input: { prefix: null, depth: 0, maxDepth: 4 }

2. Level 1 - Single Letters (A-Z)

// Each job searches for symbols starting with that letter
input: { prefix: 'A', depth: 1, maxDepth: 4 }

// If 10+ symbols found, creates: AA, AB, AC, ... AZ

3. Level 2 - Two Letters (AA-ZZ)

// Searches for symbols starting with two letters
input: { prefix: 'AA', depth: 2, maxDepth: 4 }

// If 10+ symbols found, creates: AAA, AAB, AAC, ... AAZ

4. Level 3 & 4 - Deeper Search

Continues until maxDepth is reached or fewer than 10 symbols are found.

Testing the Spider

Manual Test - Root Job

# Trigger spider search root job
curl -X POST http://localhost:3000/api/handlers/qm/operations/spider-symbol-search \
  -H "Content-Type: application/json" \
  -d '{}'

Manual Test - Specific Prefix

# Search for symbols starting with "APP"
curl -X POST http://localhost:3000/api/handlers/qm/operations/spider-symbol-search \
  -H "Content-Type: application/json" \
  -d '{"prefix": "APP", "depth": 3, "maxDepth": 4}'

Direct Symbol Search

# Search for specific symbols
curl -X POST http://localhost:3000/api/handlers/qm/operations/search-symbols \
  -H "Content-Type: application/json" \
  -d '{"query": "AAPL"}'

Optimization Features

1. Intelligent Branching: Only creates child jobs if 10+ symbols found
2. Priority Management: Deeper searches get lower priority
3. Staggered Execution: Jobs are delayed to avoid API rate limits
4. Session Management: Uses QM sessions with failure tracking
5. Caching: Results cached for 30 minutes to avoid duplicate API calls

MongoDB Collections

• qm_symbols: Stores discovered symbols with metadata
• qm_exchanges: Stores unique exchanges found during searches

Monitoring

Check logs for:

• "Spider symbol search" - Job execution
• "Created root spider jobs" - Initial A-Z creation
• "Stored symbols from spider search" - Successful symbol storage
• "Found X symbols for Y, queued Z child jobs" - Branching decisions