boki/stock-bot
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

renaming services to more suitable names

Browse source
This commit is contained in:
Boki 2025-06-21 14:02:54 -04:00
parent 3ae9de8376
commit be6afef832
69 changed files with 41 additions and 2956 deletions
Download patch file Download diff file Expand all files Collapse all files

183
apps/data-sync-service/README.md
View file

@ -1,183 +0,0 @@
# Data Sync Service
The Data Sync Service handles synchronization of raw MongoDB data to PostgreSQL master records, providing a unified data layer for the stock-bot application.
## Features
### Original Sync Manager
- Basic QM (QuoteMedia) symbol and exchange synchronization
- Simple static exchange mapping
- Manual sync triggers via REST API
### Enhanced Sync Manager ✨ NEW
- **Multi-provider support**: Syncs from EOD, Interactive Brokers, and QuoteMedia
- **Comprehensive exchange handling**: Leverages all 4 MongoDB exchange collections
- **Intelligent exchange mapping**: Dynamic mapping with fallback logic
- **Transaction safety**: Full ACID compliance with rollback on errors
- **Performance optimization**: Exchange caching for faster lookups
- **Enhanced error handling**: Detailed error tracking and reporting
## API Endpoints
### Health Check
- `GET /health` - Service health status
### Original Sync Operations
- `POST /sync/symbols` - Sync QM symbols to PostgreSQL
- `POST /sync/exchanges` - Sync QM exchanges to PostgreSQL
- `GET /sync/status` - Get basic sync status
### Enhanced Sync Operations ✨ NEW
- `POST /sync/exchanges/all?clear=true` - Comprehensive exchange sync from all providers (clear=true removes dummy data first)
- `POST /sync/symbols/:provider?clear=true` - Sync symbols from specific provider (qm, eod, ib)
- `POST /sync/clear` - Clear all PostgreSQL data (exchanges, symbols, mappings)
- `GET /sync/status/enhanced` - Get detailed sync status
- `GET /sync/stats/exchanges` - Get exchange statistics
## Data Sources
### MongoDB Collections
1. **exchanges** (34 records) - Unified exchange reference
2. **eodExchanges** (78 records) - EOD provider with currency/MIC data
3. **ibExchanges** (214 records) - Interactive Brokers with asset types
4. **qmExchanges** (25 records) - QuoteMedia exchanges
### PostgreSQL Tables
1. **master_exchanges** - Unified exchange master data
2. **master_symbols** - Symbol master records
3. **provider_symbol_mappings** - Multi-provider symbol mappings
4. **sync_status** - Synchronization tracking
## Key Improvements
### 1. Multi-Provider Exchange Sync
Instead of only syncing QM exchanges, the enhanced sync manager:
- Syncs from EOD exchanges (comprehensive global data with currencies)
- Adds IB exchanges for additional coverage (214 exchanges vs 25 in QM)
### 2. Intelligent Exchange Mapping
Replaces hard-coded mapping with dynamic resolution:
```typescript
// Before: Static mapping
const exchangeMap = { 'NASDAQ': 'NASDAQ', 'NYSE': 'NYSE' };
// After: Dynamic mapping with variations
const codeMap = {
'NASDAQ': 'NASDAQ', 'NAS': 'NASDAQ',
'NYSE': 'NYSE', 'NYQ': 'NYSE',
'LSE': 'LSE', 'LON': 'LSE', 'LN': 'LSE',
'US': 'NYSE' // EOD uses 'US' for US markets
};
```
### 3. Transaction Safety
All sync operations use database transactions:
- `BEGIN` transaction at start
- `COMMIT` on success
- `ROLLBACK` on any error
- Ensures data consistency
### 4. Performance Optimization
- Exchange cache preloaded at startup
- Reduced database queries during symbol processing
- Batch operations where possible
### 5. Enhanced Error Handling
- Detailed error logging with context
- Separate error counting in sync results
- Graceful handling of missing/invalid data
## Usage Examples
### Clear All Data and Start Fresh Exchange Sync
```bash
curl -X POST "http://localhost:3005/sync/exchanges/all?clear=true"
```
### Sync Symbols from Specific Provider
```bash
# Sync QuoteMedia symbols (clear existing symbols first)
curl -X POST "http://localhost:3005/sync/symbols/qm?clear=true"
# Sync EOD symbols
curl -X POST http://localhost:3005/sync/symbols/eod
# Sync Interactive Brokers symbols
curl -X POST http://localhost:3005/sync/symbols/ib
```
### Clear All PostgreSQL Data
```bash
curl -X POST http://localhost:3005/sync/clear
```
### Get Enhanced Status
```bash
curl http://localhost:3005/sync/status/enhanced
curl http://localhost:3005/sync/stats/exchanges
```
## Configuration
### Environment Variables
- `DATA_SYNC_SERVICE_PORT` - Service port (default: 3005)
- `NODE_ENV` - Environment mode
### Database Connections
- **MongoDB**: `mongodb://trading_admin:trading_mongo_dev@localhost:27017/stock?authSource=admin`
- **PostgreSQL**: `postgresql://trading_user:trading_pass_dev@localhost:5432/trading_bot`
## Development
### Build and Run
```bash
# Development mode
bun run dev
# Build
bun run build
# Production
bun run start
```
### Testing
```bash
# Run tests
bun test
# Start infrastructure
bun run infra:up
# Test sync operations
curl -X POST http://localhost:3005/sync/exchanges/all
curl -X POST http://localhost:3005/sync/symbols/qm
```
## Architecture
```
MongoDB Collections PostgreSQL Tables
┌─ exchanges (34) ┐ ┌─ master_exchanges
├─ eodExchanges (78) ├──▶├─ master_symbols
├─ ibExchanges (214) │ ├─ provider_symbol_mappings
└─ qmExchanges (25) ┘ └─ sync_status
│
▼
Enhanced Sync Manager
- Exchange caching
- Dynamic mapping
- Transaction safety
- Multi-provider support
```
## Migration Path
The enhanced sync manager is designed to work alongside the original sync manager:
1. **Immediate**: Use enhanced exchange sync for better coverage
2. **Phase 1**: Test enhanced symbol sync with each provider
3. **Phase 2**: Replace original sync manager when confident
4. **Phase 3**: Remove original sync manager and endpoints
Both managers can be used simultaneously during the transition period.