work on market-data-gateway

docs/core-services/market-data-gateway/README.md

@@ -0,0 +1,82 @@
+# Market Data Gateway
+
+## Overview
+The Market Data Gateway (MDG) service serves as the central hub for real-time market data processing and distribution within the stock-bot platform. It acts as the intermediary between external market data providers and internal platform services, ensuring consistent, normalized, and reliable market data delivery.
+
+## Key Features
+
+### Real-time Data Processing
+- **WebSocket Streaming**: Provides low-latency data streams for market updates
+- **Multi-source Aggregation**: Integrates data from multiple providers (Alpaca, Yahoo Finance, etc.)
+- **Normalized Data Model**: Transforms varied provider formats into a unified platform data model
+- **Subscription Management**: Allows services to subscribe to specific data streams
+
+### Data Quality Management
+- **Validation & Sanitization**: Ensures data integrity through validation rules
+- **Anomaly Detection**: Identifies unusual price movements or data issues
+- **Gap Filling**: Interpolation strategies for missing data points
+- **Data Reconciliation**: Cross-validates data from multiple sources
+
+### Performance Optimization
+- **Caching Layer**: In-memory cache for frequently accessed data
+- **Rate Limiting**: Protects against API quota exhaustion
+- **Connection Pooling**: Efficiently manages provider connections
+- **Compression**: Minimizes data transfer size for bandwidth efficiency
+
+### Operational Resilience
+- **Automatic Reconnection**: Handles provider disconnections gracefully
+- **Circuit Breaking**: Prevents cascade failures during outages
+- **Failover Mechanisms**: Switches to alternative data sources when primary sources fail
+- **Health Monitoring**: Self-reports service health metrics
+
+## Integration Points
+
+### Upstream Connections
+- Alpaca Markets API (primary data source)
+- Yahoo Finance API (secondary data source)
+- Potential future integrations with IEX, Polygon, etc.
+
+### Downstream Consumers
+- Strategy Orchestrator
+- Risk Guardian
+- Trading Dashboard
+- Data Persistence Layer
+
+## Technical Implementation
+
+### Technology Stack
+- **Runtime**: Node.js with TypeScript
+- **Messaging**: WebSockets for real-time streaming
+- **Caching**: Redis for shared cache
+- **Metrics**: Prometheus metrics for monitoring
+- **Configuration**: Environment-based with runtime updates
+
+### Architecture Pattern
+- Event-driven microservice with publisher-subscriber model
+- Horizontally scalable to handle increased data volumes
+- Stateless design with external state management
+
+## Development Guidelines
+
+### Error Handling
+- Detailed error classification and handling strategy
+- Graceful degradation during partial outages
+- Comprehensive error logging with context
+
+### Testing Strategy
+- Unit tests for data transformation logic
+- Integration tests with mock data providers
+- Performance tests for throughput capacity
+- Chaos testing for resilience verification
+
+### Observability
+- Detailed logs for troubleshooting
+- Performance metrics for optimization
+- Health checks for system monitoring
+- Tracing for request flow analysis
+
+## Future Enhancements
+- Support for options and derivatives data
+- Real-time news and sentiment integration
+- Machine learning-based data quality improvements
+- Enhanced historical data query capabilities