stock-bot/apps/core-services/market-data-gateway

Market Data Gateway - Unified Implementation

Overview

The Market Data Gateway is a unified service that consolidates real-time market data ingestion, processing, and distribution capabilities. This service has been created by merging the previous core-services and data-services market-data-gateway implementations into a single, comprehensive solution.

Architecture

Unified Design

• Single Service: Combines data ingestion, processing, and distribution in one service
• HTTP API: RESTful endpoints for configuration and data retrieval
• WebSocket Server: Real-time data streaming capabilities
• Type Safety: Full TypeScript implementation with comprehensive type definitions

Key Components

• Data Source Management: Configure and manage multiple market data sources
• Real-time Processing: Stream processing pipelines for market data
• WebSocket Streaming: Real-time data distribution to clients
• Health Monitoring: Comprehensive health checks and metrics
• Cache Management: Redis-based caching for performance optimization

Features

HTTP Endpoints

Health & Status

• GET /health - Basic health check
• GET /health/readiness - Readiness probe
• GET /health/liveness - Liveness probe
• GET /api/v1/gateway/status - Gateway status and metrics
• GET /api/v1/gateway/config - Current configuration

Data Sources

• GET /api/v1/sources - List configured data sources
• POST /api/v1/sources - Add new data source
• PUT /api/v1/sources/:sourceId - Update data source
• DELETE /api/v1/sources/:sourceId - Remove data source

Market Data

• GET /api/v1/data/tick/:symbol - Latest tick data for symbol
• GET /api/v1/data/candles/:symbol - Historical candle data
• GET /api/v1/subscriptions - List active subscriptions
• POST /api/v1/subscriptions - Create new subscription

Metrics

• GET /api/v1/metrics - System and gateway metrics

WebSocket Streaming

Connect to ws://localhost:3005/ws for real-time data streaming.

Message Types

Subscribe to symbols:

{
  "type": "subscribe",
  "symbols": ["AAPL", "GOOGL", "MSFT"]
}

Unsubscribe:

{
  "type": "unsubscribe",
  "subscriptionId": "sub_1234567890"
}

Receive tick data:

{
  "type": "tick",
  "data": {
    "symbol": "AAPL",
    "price": 150.25,
    "volume": 1000,
    "timestamp": "2025-06-03T13:01:49.638Z",
    "bid": 150.20,
    "ask": 150.30
  }
}

Configuration

The service is configured through environment variables and the GatewayConfig interface:

Environment Variables

• PORT - HTTP server port (default: 3004)
• HOST - Server host (default: 0.0.0.0)
• REDIS_HOST - Redis host (default: localhost)
• REDIS_PORT - Redis port (default: 6379)
• REDIS_PASSWORD - Redis password (optional)
• REDIS_DB - Redis database number (default: 0)
• METRICS_PORT - Metrics port (default: 9090)

Configuration Structure

interface GatewayConfig {
  server: ServerConfig;
  dataSources: DataSourceConfig[];
  processing: ProcessingConfig;
  cache: CacheConfig;
  monitoring: MonitoringConfig;
}

Development

Prerequisites

• Bun runtime
• Redis server
• TypeScript

Setup

cd apps/core-services/market-data-gateway
bun install

Development Mode

bun run dev

Build

bun run build

Testing

The service includes mock data for testing purposes. When running, it will:

• Respond to health checks
• Provide mock tick and candle data
• Accept WebSocket connections
• Send simulated real-time data every 5 seconds

Deployment

The service can be deployed using:

• Docker containers
• Kubernetes
• Direct Node.js/Bun deployment

Docker

FROM oven/bun:latest
WORKDIR /app
COPY package.json .
COPY src/ ./src/
RUN bun install
RUN bun run build
EXPOSE 3004 3005
CMD ["bun", "run", "start"]

Migration Notes

This unified implementation replaces both:

• apps/core-services/market-data-gateway (original)
• apps/data-services/market-data-gateway (duplicate)

Changes Made

1. Consolidated Architecture: Merged real-time and storage capabilities
2. Fixed Type Issues: Resolved all TypeScript compilation errors
3. Simplified Configuration: Aligned with GatewayConfig interface
4. Working WebSocket: Functional real-time streaming
5. Comprehensive API: Full REST API implementation
6. Mock Data: Testing capabilities with simulated data

Removed Duplicates

• Removed apps/data-services/market-data-gateway directory
• Consolidated type definitions
• Unified configuration structure

Future Enhancements

1. Real Data Sources: Replace mock data with actual market data feeds
2. Advanced Processing: Implement complex processing pipelines
3. Persistence Layer: Add database storage for historical data
4. Authentication: Add API authentication and authorization
5. Rate Limiting: Implement request rate limiting
6. Monitoring: Enhanced metrics and alerting
7. Load Balancing: Support for horizontal scaling

Status

✅ COMPLETED: TypeScript compilation errors resolved ✅ COMPLETED: Unified service architecture ✅ COMPLETED: Working HTTP and WebSocket servers ✅ COMPLETED: Mock data implementation ✅ COMPLETED: Health and metrics endpoints ✅ COMPLETED: Duplicate service removal

The market data gateway merge is now complete and the service is fully operational.