work on market-data-gateway

2025-06-03 09:57:11 -04:00 · 2025-06-03 09:57:11 -04:00 · b957fb99aa
commit b957fb99aa
parent 405b818c86
87 changed files with 7979 additions and 99 deletions
--- a/apps/core-services/market-data-gateway/README.md
+++ b/apps/core-services/market-data-gateway/README.md
@ -0,0 +1,196 @@
+# Market Data Gateway - Unified Implementation
+
+## Overview
+
+The Market Data Gateway is a unified service that consolidates real-time market data ingestion, processing, and distribution capabilities. This service has been created by merging the previous core-services and data-services market-data-gateway implementations into a single, comprehensive solution.
+
+## Architecture
+
+### Unified Design
+- **Single Service**: Combines data ingestion, processing, and distribution in one service
+- **HTTP API**: RESTful endpoints for configuration and data retrieval
+- **WebSocket Server**: Real-time data streaming capabilities
+- **Type Safety**: Full TypeScript implementation with comprehensive type definitions
+
+### Key Components
+- **Data Source Management**: Configure and manage multiple market data sources
+- **Real-time Processing**: Stream processing pipelines for market data
+- **WebSocket Streaming**: Real-time data distribution to clients
+- **Health Monitoring**: Comprehensive health checks and metrics
+- **Cache Management**: Redis-based caching for performance optimization
+
+## Features
+
+### HTTP Endpoints
+
+#### Health & Status
+- `GET /health` - Basic health check
+- `GET /health/readiness` - Readiness probe
+- `GET /health/liveness` - Liveness probe
+- `GET /api/v1/gateway/status` - Gateway status and metrics
+- `GET /api/v1/gateway/config` - Current configuration
+
+#### Data Sources
+- `GET /api/v1/sources` - List configured data sources
+- `POST /api/v1/sources` - Add new data source
+- `PUT /api/v1/sources/:sourceId` - Update data source
+- `DELETE /api/v1/sources/:sourceId` - Remove data source
+
+#### Market Data
+- `GET /api/v1/data/tick/:symbol` - Latest tick data for symbol
+- `GET /api/v1/data/candles/:symbol` - Historical candle data
+- `GET /api/v1/subscriptions` - List active subscriptions
+- `POST /api/v1/subscriptions` - Create new subscription
+
+#### Metrics
+- `GET /api/v1/metrics` - System and gateway metrics
+
+### WebSocket Streaming
+
+Connect to `ws://localhost:3005/ws` for real-time data streaming.
+
+#### Message Types
+
+**Subscribe to symbols:**
+```json
+{
+  "type": "subscribe",
+  "symbols": ["AAPL", "GOOGL", "MSFT"]
+}
+```
+
+**Unsubscribe:**
+```json
+{
+  "type": "unsubscribe",
+  "subscriptionId": "sub_1234567890"
+}
+```
+
+**Receive tick data:**
+```json
+{
+  "type": "tick",
+  "data": {
+    "symbol": "AAPL",
+    "price": 150.25,
+    "volume": 1000,
+    "timestamp": "2025-06-03T13:01:49.638Z",
+    "bid": 150.20,
+    "ask": 150.30
+  }
+}
+```
+
+## Configuration
+
+The service is configured through environment variables and the `GatewayConfig` interface:
+
+### Environment Variables
+- `PORT` - HTTP server port (default: 3004)
+- `HOST` - Server host (default: 0.0.0.0)
+- `REDIS_HOST` - Redis host (default: localhost)
+- `REDIS_PORT` - Redis port (default: 6379)
+- `REDIS_PASSWORD` - Redis password (optional)
+- `REDIS_DB` - Redis database number (default: 0)
+- `METRICS_PORT` - Metrics port (default: 9090)
+
+### Configuration Structure
+```typescript
+interface GatewayConfig {
+  server: ServerConfig;
+  dataSources: DataSourceConfig[];
+  processing: ProcessingConfig;
+  cache: CacheConfig;
+  monitoring: MonitoringConfig;
+}
+```
+
+## Development
+
+### Prerequisites
+- Bun runtime
+- Redis server
+- TypeScript
+
+### Setup
+```bash
+cd apps/core-services/market-data-gateway
+bun install
+```
+
+### Development Mode
+```bash
+bun run dev
+```
+
+### Build
+```bash
+bun run build
+```
+
+### Testing
+The service includes mock data for testing purposes. When running, it will:
+- Respond to health checks
+- Provide mock tick and candle data
+- Accept WebSocket connections
+- Send simulated real-time data every 5 seconds
+
+## Deployment
+
+The service can be deployed using:
+- Docker containers
+- Kubernetes
+- Direct Node.js/Bun deployment
+
+### Docker
+```dockerfile
+FROM oven/bun:latest
+WORKDIR /app
+COPY package.json .
+COPY src/ ./src/
+RUN bun install
+RUN bun run build
+EXPOSE 3004 3005
+CMD ["bun", "run", "start"]
+```
+
+## Migration Notes
+
+This unified implementation replaces both:
+- `apps/core-services/market-data-gateway` (original)
+- `apps/data-services/market-data-gateway` (duplicate)
+
+### Changes Made
+1. **Consolidated Architecture**: Merged real-time and storage capabilities
+2. **Fixed Type Issues**: Resolved all TypeScript compilation errors
+3. **Simplified Configuration**: Aligned with `GatewayConfig` interface
+4. **Working WebSocket**: Functional real-time streaming
+5. **Comprehensive API**: Full REST API implementation
+6. **Mock Data**: Testing capabilities with simulated data
+
+### Removed Duplicates
+- Removed `apps/data-services/market-data-gateway` directory
+- Consolidated type definitions
+- Unified configuration structure
+
+## Future Enhancements
+
+1. **Real Data Sources**: Replace mock data with actual market data feeds
+2. **Advanced Processing**: Implement complex processing pipelines
+3. **Persistence Layer**: Add database storage for historical data
+4. **Authentication**: Add API authentication and authorization
+5. **Rate Limiting**: Implement request rate limiting
+6. **Monitoring**: Enhanced metrics and alerting
+7. **Load Balancing**: Support for horizontal scaling
+
+## Status
+
+✅ **COMPLETED**: TypeScript compilation errors resolved
+✅ **COMPLETED**: Unified service architecture
+✅ **COMPLETED**: Working HTTP and WebSocket servers
+✅ **COMPLETED**: Mock data implementation
+✅ **COMPLETED**: Health and metrics endpoints
+✅ **COMPLETED**: Duplicate service removal
+
+The market data gateway merge is now complete and the service is fully operational.