stock-bot/apps/data-sync-service/README.md

# Data Sync Service

The Data Sync Service handles synchronization of raw MongoDB data to PostgreSQL master records, providing a unified data layer for the stock-bot application.

## Features

### Original Sync Manager
- Basic QM (QuoteMedia) symbol and exchange synchronization
- Simple static exchange mapping
- Manual sync triggers via REST API

### Enhanced Sync Manager ✨ NEW
- **Multi-provider support**: Syncs from EOD, Interactive Brokers, and QuoteMedia
- **Comprehensive exchange handling**: Leverages all 4 MongoDB exchange collections
- **Intelligent exchange mapping**: Dynamic mapping with fallback logic
- **Transaction safety**: Full ACID compliance with rollback on errors
- **Performance optimization**: Exchange caching for faster lookups
- **Enhanced error handling**: Detailed error tracking and reporting

## API Endpoints

### Health Check
- `GET /health` - Service health status

### Original Sync Operations
- `POST /sync/symbols` - Sync QM symbols to PostgreSQL
- `POST /sync/exchanges` - Sync QM exchanges to PostgreSQL
- `GET /sync/status` - Get basic sync status

### Enhanced Sync Operations ✨ NEW
- `POST /sync/exchanges/all?clear=true` - Comprehensive exchange sync from all providers (clear=true removes dummy data first)
- `POST /sync/symbols/:provider?clear=true` - Sync symbols from specific provider (qm, eod, ib)
- `POST /sync/clear` - Clear all PostgreSQL data (exchanges, symbols, mappings)
- `GET /sync/status/enhanced` - Get detailed sync status
- `GET /sync/stats/exchanges` - Get exchange statistics

## Data Sources

### MongoDB Collections
1. **exchanges** (34 records) - Unified exchange reference
2. **eodExchanges** (78 records) - EOD provider with currency/MIC data
3. **ibExchanges** (214 records) - Interactive Brokers with asset types
4. **qmExchanges** (25 records) - QuoteMedia exchanges

### PostgreSQL Tables
1. **master_exchanges** - Unified exchange master data
2. **master_symbols** - Symbol master records
3. **provider_symbol_mappings** - Multi-provider symbol mappings
4. **sync_status** - Synchronization tracking

## Key Improvements

### 1. Multi-Provider Exchange Sync
Instead of only syncing QM exchanges, the enhanced sync manager:
- Syncs from EOD exchanges (comprehensive global data with currencies)
- Adds IB exchanges for additional coverage (214 exchanges vs 25 in QM)

### 2. Intelligent Exchange Mapping
Replaces hard-coded mapping with dynamic resolution:
```typescript
// Before: Static mapping
const exchangeMap = { 'NASDAQ': 'NASDAQ', 'NYSE': 'NYSE' };

// After: Dynamic mapping with variations
const codeMap = {
  'NASDAQ': 'NASDAQ', 'NAS': 'NASDAQ',
  'NYSE': 'NYSE', 'NYQ': 'NYSE',
  'LSE': 'LSE', 'LON': 'LSE', 'LN': 'LSE',
  'US': 'NYSE' // EOD uses 'US' for US markets
};
```

### 3. Transaction Safety
All sync operations use database transactions:
- `BEGIN` transaction at start
- `COMMIT` on success
- `ROLLBACK` on any error
- Ensures data consistency

### 4. Performance Optimization
- Exchange cache preloaded at startup
- Reduced database queries during symbol processing
- Batch operations where possible

### 5. Enhanced Error Handling
- Detailed error logging with context
- Separate error counting in sync results
- Graceful handling of missing/invalid data

## Usage Examples

### Clear All Data and Start Fresh Exchange Sync
```bash
curl -X POST "http://localhost:3005/sync/exchanges/all?clear=true"
```

### Sync Symbols from Specific Provider
```bash
# Sync QuoteMedia symbols (clear existing symbols first)
curl -X POST "http://localhost:3005/sync/symbols/qm?clear=true"

# Sync EOD symbols
curl -X POST http://localhost:3005/sync/symbols/eod

# Sync Interactive Brokers symbols
curl -X POST http://localhost:3005/sync/symbols/ib
```

### Clear All PostgreSQL Data
```bash
curl -X POST http://localhost:3005/sync/clear
```

### Get Enhanced Status
```bash
curl http://localhost:3005/sync/status/enhanced
curl http://localhost:3005/sync/stats/exchanges
```

## Configuration

### Environment Variables
- `DATA_SYNC_SERVICE_PORT` - Service port (default: 3005)
- `NODE_ENV` - Environment mode

### Database Connections
- **MongoDB**: `mongodb://trading_admin:trading_mongo_dev@localhost:27017/stock?authSource=admin`
- **PostgreSQL**: `postgresql://trading_user:trading_pass_dev@localhost:5432/trading_bot`

## Development

### Build and Run
```bash
# Development mode
bun run dev

# Build
bun run build

# Production
bun run start
```

### Testing
```bash
# Run tests
bun test

# Start infrastructure
bun run infra:up

# Test sync operations
curl -X POST http://localhost:3005/sync/exchanges/all
curl -X POST http://localhost:3005/sync/symbols/qm
```

## Architecture

```
MongoDB Collections          PostgreSQL Tables
┌─ exchanges (34)        ┐   ┌─ master_exchanges
├─ eodExchanges (78)     ├──▶├─ master_symbols
├─ ibExchanges (214)     │   ├─ provider_symbol_mappings
└─ qmExchanges (25)      ┘   └─ sync_status
        │
        ▼
Enhanced Sync Manager
- Exchange caching
- Dynamic mapping
- Transaction safety
- Multi-provider support
```

## Migration Path

The enhanced sync manager is designed to work alongside the original sync manager:

1. **Immediate**: Use enhanced exchange sync for better coverage
2. **Phase 1**: Test enhanced symbol sync with each provider
3. **Phase 2**: Replace original sync manager when confident
4. **Phase 3**: Remove original sync manager and endpoints

Both managers can be used simultaneously during the transition period.