stock-bot/apps/data-sync-service

Data Sync Service

The Data Sync Service handles synchronization of raw MongoDB data to PostgreSQL master records, providing a unified data layer for the stock-bot application.

Features

Original Sync Manager

• Basic QM (QuoteMedia) symbol and exchange synchronization
• Simple static exchange mapping
• Manual sync triggers via REST API

Enhanced Sync Manager ✨ NEW

• Multi-provider support: Syncs from EOD, Interactive Brokers, and QuoteMedia
• Comprehensive exchange handling: Leverages all 4 MongoDB exchange collections
• Intelligent exchange mapping: Dynamic mapping with fallback logic
• Transaction safety: Full ACID compliance with rollback on errors
• Performance optimization: Exchange caching for faster lookups
• Enhanced error handling: Detailed error tracking and reporting

API Endpoints

Health Check

• GET /health - Service health status

Original Sync Operations

• POST /sync/symbols - Sync QM symbols to PostgreSQL
• POST /sync/exchanges - Sync QM exchanges to PostgreSQL
• GET /sync/status - Get basic sync status

Enhanced Sync Operations ✨ NEW

• POST /sync/exchanges/all?clear=true - Comprehensive exchange sync from all providers (clear=true removes dummy data first)
• POST /sync/symbols/:provider?clear=true - Sync symbols from specific provider (qm, eod, ib)
• POST /sync/clear - Clear all PostgreSQL data (exchanges, symbols, mappings)
• GET /sync/status/enhanced - Get detailed sync status
• GET /sync/stats/exchanges - Get exchange statistics

Data Sources

MongoDB Collections

1. exchanges (34 records) - Unified exchange reference
2. eodExchanges (78 records) - EOD provider with currency/MIC data
3. ibExchanges (214 records) - Interactive Brokers with asset types
4. qmExchanges (25 records) - QuoteMedia exchanges

PostgreSQL Tables

1. master_exchanges - Unified exchange master data
2. master_symbols - Symbol master records
3. provider_symbol_mappings - Multi-provider symbol mappings
4. sync_status - Synchronization tracking

Key Improvements

1. Multi-Provider Exchange Sync

Instead of only syncing QM exchanges, the enhanced sync manager:

• Syncs from unified exchanges collection first (primary source)
• Enriches with EOD exchanges (comprehensive global data with currencies)
• Adds IB exchanges for additional coverage (214 exchanges vs 25 in QM)

2. Intelligent Exchange Mapping

Replaces hard-coded mapping with dynamic resolution:

// Before: Static mapping
const exchangeMap = { 'NASDAQ': 'NASDAQ', 'NYSE': 'NYSE' };

// After: Dynamic mapping with variations
const codeMap = {
  'NASDAQ': 'NASDAQ', 'NAS': 'NASDAQ',
  'NYSE': 'NYSE', 'NYQ': 'NYSE',
  'LSE': 'LSE', 'LON': 'LSE', 'LN': 'LSE',
  'US': 'NYSE' // EOD uses 'US' for US markets
};

3. Transaction Safety

All sync operations use database transactions:

• BEGIN transaction at start
• COMMIT on success
• ROLLBACK on any error
• Ensures data consistency

4. Performance Optimization

• Exchange cache preloaded at startup
• Reduced database queries during symbol processing
• Batch operations where possible

5. Enhanced Error Handling

• Detailed error logging with context
• Separate error counting in sync results
• Graceful handling of missing/invalid data

Usage Examples

Clear All Data and Start Fresh Exchange Sync

curl -X POST "http://localhost:3005/sync/exchanges/all?clear=true"

Sync Symbols from Specific Provider

# Sync QuoteMedia symbols (clear existing symbols first)
curl -X POST "http://localhost:3005/sync/symbols/qm?clear=true"

# Sync EOD symbols
curl -X POST http://localhost:3005/sync/symbols/eod

# Sync Interactive Brokers symbols
curl -X POST http://localhost:3005/sync/symbols/ib

Clear All PostgreSQL Data

curl -X POST http://localhost:3005/sync/clear

Get Enhanced Status

curl http://localhost:3005/sync/status/enhanced
curl http://localhost:3005/sync/stats/exchanges

Configuration

Environment Variables

• DATA_SYNC_SERVICE_PORT - Service port (default: 3005)
• NODE_ENV - Environment mode

Database Connections

• MongoDB: mongodb://trading_admin:trading_mongo_dev@localhost:27017/stock?authSource=admin
• PostgreSQL: postgresql://trading_user:trading_pass_dev@localhost:5432/trading_bot

Development

Build and Run

# Development mode
bun run dev

# Build
bun run build

# Production
bun run start

Testing

# Run tests
bun test

# Start infrastructure
bun run infra:up

# Test sync operations
curl -X POST http://localhost:3005/sync/exchanges/all
curl -X POST http://localhost:3005/sync/symbols/qm

Architecture

MongoDB Collections          PostgreSQL Tables
┌─ exchanges (34)        ┐   ┌─ master_exchanges
├─ eodExchanges (78)     ├──▶├─ master_symbols
├─ ibExchanges (214)     │   ├─ provider_symbol_mappings
└─ qmExchanges (25)      ┘   └─ sync_status
        │
        ▼
Enhanced Sync Manager
- Exchange caching
- Dynamic mapping
- Transaction safety
- Multi-provider support

Migration Path

The enhanced sync manager is designed to work alongside the original sync manager:

1. Immediate: Use enhanced exchange sync for better coverage
2. Phase 1: Test enhanced symbol sync with each provider
3. Phase 2: Replace original sync manager when confident
4. Phase 3: Remove original sync manager and endpoints

Both managers can be used simultaneously during the transition period.