Bojan Kucera 3d9afd711e renamed http-client to http and fixed tests

2025-06-07 13:20:58 -04:00

26 KiB

Raw Blame History

Stock Trading Bot - Simplified Combined Architecture

Overview

A TypeScript-based simplified microservices architecture for automated stock trading with integrated multi-mode backtesting capabilities. The system combines live trading and backtesting functionality into unified services while supporting live, event-driven, and vectorized backtesting modes.

Architecture Diagram

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Data Service   │    │Processing Service│   │Strategy Service │
│                 │    │                 │    │                 │
│ • Live APIs     │────▶│ • Indicators    │────▶│ • Live Mode     │
│ • Historical    │    │ • Processing    │    │ • Event Mode    │
│ • QuestDB       │    │ • Validation    │    │ • Vector Mode   │
│ • Unified API   │    │ • Utils Lib     │    │ • Backtesting   │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │              ┌─────────────────┐              │
         │              │   Event Bus     │              │
         └──────────────▶│  (Dragonfly)    │◀─────────────┘
                        └─────────────────┘
                                 │
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│Execution Service│    │Portfolio Service│    │   Dashboard     │
│                 │    │                 │    │                 │
│ • Live Brokers  │    │ • Positions     │    │ • Angular UI    │
│ • Simulation    │    │ • Risk Mgmt     │    │ • Real-time     │
│ • Mode Switch   │    │ • P&L Tracking  │    │ • Backtest UI   │
│ • Unified API   │    │ • Performance   │    │ • Reports       │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Simplified Service Structure

6 Core Services (Instead of 12)

stock-bot/
├── apps/
│   ├── data-service/             # Combined data ingestion & historical
│   │   ├── src/
│   │   │   ├── providers/        # Live & historical data providers
│   │   │   │   ├── live/         # Real-time APIs (WebSocket, REST)
│   │   │   │   ├── historical/   # Historical data (Yahoo, Alpha Vantage)
│   │   │   │   └── unified.ts    # Unified data interface
│   │   │   ├── storage/          # QuestDB time-series storage
│   │   │   ├── services/
│   │   │   └── index.ts
│   │   └── package.json
│   │
│   ├── processing-service/       # Combined processing & indicators
│   │   ├── src/
│   │   │   ├── indicators/       # Technical indicators (uses @stock-bot/utils)
│   │   │   ├── processors/       # Data processing pipeline
│   │   │   ├── vectorized/       # Vectorized calculations
│   │   │   ├── services/
│   │   │   └── index.ts
│   │   └── package.json
│   │
│   ├── strategy-service/         # Combined strategy & backtesting
│   │   ├── src/
│   │   │   ├── strategies/       # Strategy implementations
│   │   │   ├── backtesting/      # Multi-mode backtesting engine
│   │   │   │   ├── modes/        # Backtesting modes
│   │   │   │   │   ├── live-mode.ts      # Live trading mode
│   │   │   │   │   ├── event-mode.ts     # Event-driven backtest
│   │   │   │   │   └── vector-mode.ts    # Vectorized backtest
│   │   │   │   ├── engines/      # Execution engines
│   │   │   │   │   ├── event-engine.ts   # Event-based simulation
│   │   │   │   │   ├── vector-engine.ts  # Vectorized calculations
│   │   │   │   │   └── hybrid-engine.ts  # Combined validation
│   │   │   │   ├── simulator.ts  # Market simulator
│   │   │   │   ├── runner.ts     # Backtest orchestrator
│   │   │   │   └── metrics.ts    # Performance analysis
│   │   │   ├── live/             # Live strategy execution
│   │   │   ├── framework/        # Strategy framework
│   │   │   │   ├── base-strategy.ts
│   │   │   │   ├── execution-mode.ts
│   │   │   │   └── mode-factory.ts
│   │   │   └── index.ts
│   │   └── package.json
│   │
│   ├── execution-service/        # Combined order execution & simulation
│   │   ├── src/
│   │   │   ├── brokers/          # Live broker adapters
│   │   │   ├── simulation/       # Simulated execution
│   │   │   ├── unified/          # Unified execution interface
│   │   │   │   ├── executor.ts   # Abstract executor
│   │   │   │   ├── live-executor.ts
│   │   │   │   ├── sim-executor.ts
│   │   │   │   └── vector-executor.ts
│   │   │   └── index.ts
│   │   └── package.json
│   │
│   ├── portfolio-service/        # Combined portfolio & risk management
│   │   ├── src/
│   │   │   ├── portfolio/        # Portfolio tracking
│   │   │   ├── risk/             # Risk management (uses @stock-bot/utils)
│   │   │   ├── positions/        # Position management
│   │   │   ├── performance/      # Performance tracking
│   │   │   └── index.ts
│   │   └── package.json
│   │
│   └── dashboard/                # Combined API & reporting
│       ├── src/
│       │   ├── api/              # REST API
│       │   ├── web/              # Web interface (Angular)
│       │   ├── reports/          # Report generation
│       │   ├── websockets/       # Real-time updates
│       │   └── index.ts
│       └── package.json
│
├── libs/                         # ✅ Your existing shared libraries
│   ├── config/                   # ✅ Environment configuration
│   ├── http/                     # ✅ HTTP utilities
│   ├── logger/                   # ✅ Loki-integrated logging
│   ├── mongodb-client/           # ✅ MongoDB operations
│   ├── postgres-client/          # ✅ PostgreSQL operations
│   ├── questdb-client/           # ✅ Time-series data
│   ├── types/                    # ✅ Shared TypeScript types
│   ├── utils/                    # ✅ Calculations & utilities
│   ├── event-bus/                # 🆕 Dragonfly event system
│   ├── strategy-engine/          # 🆕 Strategy framework
│   ├── vector-engine/            # 🆕 Vectorized calculations
│   └── data-frame/               # 🆕 DataFrame operations

Multi-Mode Backtesting Architecture

1. Execution Mode Framework

export abstract class ExecutionMode {
  protected logger = createLogger(this.constructor.name);
  protected config = new ServiceConfig();
  
  abstract name: string;
  abstract executeOrder(order: Order): Promise<OrderResult>;
  abstract getCurrentTime(): Date;
  abstract getMarketData(symbol: string): Promise<MarketData>;
  abstract publishEvent(event: string, data: any): Promise<void>;
}

export enum BacktestMode {
  LIVE = 'live',
  EVENT_DRIVEN = 'event-driven',
  VECTORIZED = 'vectorized',
  HYBRID = 'hybrid'
}

2. Live Trading Mode

export class LiveMode extends ExecutionMode {
  name = 'live';
  private broker = new BrokerClient(this.config.getBrokerConfig());
  private eventBus = new EventBus();
  
  async executeOrder(order: Order): Promise<OrderResult> {
    this.logger.info('Executing live order', { orderId: order.id });
    
    // Execute via real broker
    const result = await this.broker.placeOrder(order);
    
    // Publish to event bus
    await this.eventBus.publish('order.executed', result);
    
    return result;
  }
  
  getCurrentTime(): Date {
    return new Date(); // Real time
  }
  
  async getMarketData(symbol: string): Promise<MarketData> {
    // Get live market data
    return await this.marketDataService.getLiveData(symbol);
  }
  
  async publishEvent(event: string, data: any): Promise<void> {
    await this.eventBus.publish(event, data);
  }
}

3. Event-Driven Backtesting Mode

export class EventBacktestMode extends ExecutionMode {
  name = 'event-driven';
  private simulator = new MarketSimulator();
  private eventBus = new InMemoryEventBus(); // In-memory for simulation
  private simulationTime: Date;
  private historicalData: Map<string, MarketData[]>;
  
  constructor(private config: BacktestConfig) {
    super();
    this.simulationTime = config.startDate;
  }
  
  async executeOrder(order: Order): Promise<OrderResult> {
    this.logger.debug('Simulating order execution', { 
      orderId: order.id, 
      simulationTime: this.simulationTime 
    });
    
    // Realistic order simulation with slippage, fees
    const result = await this.simulator.executeOrder(order, {
      currentTime: this.simulationTime,
      marketData: await this.getMarketData(order.symbol),
      slippageModel: this.config.slippageModel,
      commissionModel: this.config.commissionModel
    });
    
    // Publish to simulation event bus
    await this.eventBus.publish('order.executed', result);
    
    return result;
  }
  
  getCurrentTime(): Date {
    return this.simulationTime;
  }
  
  async getMarketData(symbol: string): Promise<MarketData> {
    const data = this.historicalData.get(symbol) || [];
    return data.find(d => d.timestamp <= this.simulationTime) || null;
  }
  
  async publishEvent(event: string, data: any): Promise<void> {
    await this.eventBus.publish(event, data);
  }
  
  // Progress simulation time
  advanceTime(newTime: Date): void {
    this.simulationTime = newTime;
  }
}

4. Vectorized Backtesting Mode

export class VectorBacktestMode extends ExecutionMode {
  name = 'vectorized';
  private dataFrame: DataFrame;
  private currentIndex: number = 0;
  
  constructor(private config: VectorBacktestConfig) {
    super();
    this.dataFrame = new DataFrame(config.historicalData);
  }
  
  // Vectorized execution - processes entire dataset at once
  async executeVectorizedBacktest(strategy: VectorizedStrategy): Promise<BacktestResult> {
    const startTime = Date.now();
    
    this.logger.info('Starting vectorized backtest', { 
      strategy: strategy.name,
      dataPoints: this.dataFrame.length 
    });
    
    // Generate all signals at once using your utils library
    const signals = this.generateVectorizedSignals(strategy);
    
    // Calculate performance metrics vectorized
    const performance = this.calculateVectorizedPerformance(signals);
    
    // Apply trading costs if specified
    if (this.config.tradingCosts) {
      this.applyTradingCosts(performance, signals);
    }
    
    const executionTime = Date.now() - startTime;
    
    this.logger.info('Vectorized backtest completed', { 
      executionTime,
      totalReturn: performance.totalReturn,
      sharpeRatio: performance.sharpeRatio 
    });
    
    return {
      mode: 'vectorized',
      strategy: strategy.name,
      performance,
      executionTime,
      signals
    };
  }
  
  private generateVectorizedSignals(strategy: VectorizedStrategy): DataFrame {
    const prices = this.dataFrame.get('close');
    
    // Use your existing technical indicators from @stock-bot/utils
    const indicators = {
      sma20: sma(prices, 20),
      sma50: sma(prices, 50),
      rsi: rsi(prices, 14),
      macd: macd(prices)
    };
    
    // Generate position signals vectorized
    const positions = strategy.generatePositions(this.dataFrame, indicators);
    
    return new DataFrame({
      ...this.dataFrame.toObject(),
      ...indicators,
      positions
    });
  }
  
  private calculateVectorizedPerformance(signals: DataFrame): PerformanceMetrics {
    const prices = signals.get('close');
    const positions = signals.get('positions');
    
    // Calculate returns vectorized
    const returns = prices.slice(1).map((price, i) => 
      (price - prices[i]) / prices[i]
    );
    
    // Strategy returns = position[t-1] * market_return[t]
    const strategyReturns = returns.map((ret, i) => 
      (positions[i] || 0) * ret
    );
    
    // Use your existing performance calculation utilities
    return {
      totalReturn: calculateTotalReturn(strategyReturns),
      sharpeRatio: calculateSharpeRatio(strategyReturns),
      maxDrawdown: calculateMaxDrawdown(strategyReturns),
      volatility: calculateVolatility(strategyReturns),
      winRate: calculateWinRate(strategyReturns)
    };
  }
  
  // Standard interface methods (not used in vectorized mode)
  async executeOrder(order: Order): Promise<OrderResult> {
    throw new Error('Use executeVectorizedBacktest for vectorized mode');
  }
  
  getCurrentTime(): Date {
    return this.dataFrame.getTimestamp(this.currentIndex);
  }
  
  async getMarketData(symbol: string): Promise<MarketData> {
    return this.dataFrame.getRow(this.currentIndex);
  }
  
  async publishEvent(event: string, data: any): Promise<void> {
    // No-op for vectorized mode
  }
}

5. Hybrid Validation Mode

export class HybridBacktestMode extends ExecutionMode {
  name = 'hybrid';
  private eventMode: EventBacktestMode;
  private vectorMode: VectorBacktestMode;
  
  constructor(config: BacktestConfig) {
    super();
    this.eventMode = new EventBacktestMode(config);
    this.vectorMode = new VectorBacktestMode(config);
  }
  
  async validateStrategy(
    strategy: BaseStrategy,
    tolerance: number = 0.001
  ): Promise<ValidationResult> {
    
    this.logger.info('Starting hybrid validation', { 
      strategy: strategy.name,
      tolerance 
    });
    
    // Run vectorized backtest (fast)
    const vectorResult = await this.vectorMode.executeVectorizedBacktest(
      strategy as VectorizedStrategy
    );
    
    // Run event-driven backtest (realistic)
    const eventResult = await this.runEventBacktest(strategy);
    
    // Compare results
    const performanceDiff = Math.abs(
      vectorResult.performance.totalReturn - 
      eventResult.performance.totalReturn
    );
    
    const isValid = performanceDiff < tolerance;
    
    this.logger.info('Hybrid validation completed', { 
      isValid,
      performanceDifference: performanceDiff,
      recommendation: isValid ? 'vectorized' : 'event-driven'
    });
    
    return {
      isValid,
      performanceDifference: performanceDiff,
      vectorizedResult: vectorResult,
      eventResult,
      recommendation: isValid ? 
        'Vectorized results are reliable for this strategy' :
        'Use event-driven backtesting for accurate results'
    };
  }
  
  // Standard interface methods delegate to event mode
  async executeOrder(order: Order): Promise<OrderResult> {
    return await this.eventMode.executeOrder(order);
  }
  
  getCurrentTime(): Date {
    return this.eventMode.getCurrentTime();
  }
  
  async getMarketData(symbol: string): Promise<MarketData> {
    return await this.eventMode.getMarketData(symbol);
  }
  
  async publishEvent(event: string, data: any): Promise<void> {
    await this.eventMode.publishEvent(event, data);
  }
}

Unified Strategy Implementation

Base Strategy Framework

export abstract class BaseStrategy {
  protected mode: ExecutionMode;
  protected logger = createLogger(this.constructor.name);
  
  abstract name: string;
  abstract parameters: Record<string, any>;
  
  constructor(mode: ExecutionMode) {
    this.mode = mode;
  }
  
  // Works identically across all modes
  abstract onPriceUpdate(data: PriceData): Promise<void>;
  abstract onIndicatorUpdate(data: IndicatorData): Promise<void>;
  
  protected async emitSignal(signal: TradeSignal): Promise<void> {
    this.logger.debug('Emitting trade signal', { signal });
    
    // Mode handles whether this is live, simulated, or vectorized
    const order = this.createOrder(signal);
    const result = await this.mode.executeOrder(order);
    
    await this.mode.publishEvent('trade.executed', {
      signal,
      order,
      result,
      timestamp: this.mode.getCurrentTime()
    });
  }
  
  private createOrder(signal: TradeSignal): Order {
    return {
      id: generateId(),
      symbol: signal.symbol,
      side: signal.action,
      quantity: signal.quantity,
      type: 'market',
      timestamp: this.mode.getCurrentTime()
    };
  }
}

// Vectorized strategy interface
export interface VectorizedStrategy {
  name: string;
  parameters: Record<string, any>;
  generatePositions(data: DataFrame, indicators: any): number[];
}

Example Strategy Implementation

export class SMAStrategy extends BaseStrategy implements VectorizedStrategy {
  name = 'SMA-Crossover';
  parameters = { fastPeriod: 10, slowPeriod: 20 };
  
  private fastSMA: number[] = [];
  private slowSMA: number[] = [];
  
  async onPriceUpdate(data: PriceData): Promise<void> {
    // Same logic for live, event-driven, and hybrid modes
    this.fastSMA.push(data.close);
    this.slowSMA.push(data.close);
    
    if (this.fastSMA.length > this.parameters.fastPeriod) {
      this.fastSMA.shift();
    }
    if (this.slowSMA.length > this.parameters.slowPeriod) {
      this.slowSMA.shift();
    }
    
    if (this.fastSMA.length === this.parameters.fastPeriod && 
        this.slowSMA.length === this.parameters.slowPeriod) {
      
      const fastAvg = sma(this.fastSMA, this.parameters.fastPeriod)[0];
      const slowAvg = sma(this.slowSMA, this.parameters.slowPeriod)[0];
      
      if (fastAvg > slowAvg) {
        await this.emitSignal({
          symbol: data.symbol,
          action: 'BUY',
          quantity: 100,
          confidence: 0.8
        });
      } else if (fastAvg < slowAvg) {
        await this.emitSignal({
          symbol: data.symbol,
          action: 'SELL',
          quantity: 100,
          confidence: 0.8
        });
      }
    }
  }
  
  async onIndicatorUpdate(data: IndicatorData): Promise<void> {
    // Handle pre-calculated indicators
  }
  
  // Vectorized implementation for fast backtesting
  generatePositions(data: DataFrame, indicators: any): number[] {
    const { sma20: fastSMA, sma50: slowSMA } = indicators;
    
    return fastSMA.map((fast, i) => {
      const slow = slowSMA[i];
      if (isNaN(fast) || isNaN(slow)) return 0;
      
      // Long when fast > slow, short when fast < slow
      return fast > slow ? 1 : (fast < slow ? -1 : 0);
    });
  }
}

Mode Factory and Service Integration

Mode Factory

export class ModeFactory {
  static create(mode: BacktestMode, config: any): ExecutionMode {
    switch (mode) {
      case BacktestMode.LIVE:
        return new LiveMode();
      case BacktestMode.EVENT_DRIVEN:
        return new EventBacktestMode(config);
      case BacktestMode.VECTORIZED:
        return new VectorBacktestMode(config);
      case BacktestMode.HYBRID:
        return new HybridBacktestMode(config);
      default:
        throw new Error(`Unknown mode: ${mode}`);
    }
  }
}

Strategy Service Integration

export class StrategyService {
  private logger = createLogger('strategy-service');
  
  async runStrategy(
    strategyName: string,
    mode: BacktestMode,
    config: any
  ): Promise<any> {
    
    const executionMode = ModeFactory.create(mode, config);
    const strategy = await this.loadStrategy(strategyName, executionMode);
    
    this.logger.info('Starting strategy execution', { 
      strategy: strategyName,
      mode,
      config 
    });
    
    switch (mode) {
      case BacktestMode.LIVE:
        return await this.runLiveStrategy(strategy);
        
      case BacktestMode.EVENT_DRIVEN:
        return await this.runEventBacktest(strategy, config);
        
      case BacktestMode.VECTORIZED:
        return await (executionMode as VectorBacktestMode)
          .executeVectorizedBacktest(strategy as VectorizedStrategy);
        
      case BacktestMode.HYBRID:
        return await (executionMode as HybridBacktestMode)
          .validateStrategy(strategy, config.tolerance);
        
      default:
        throw new Error(`Unsupported mode: ${mode}`);
    }
  }
  
  async optimizeStrategy(
    strategyName: string,
    parameterGrid: Record<string, any[]>,
    config: BacktestConfig
  ): Promise<OptimizationResult[]> {
    
    const results: OptimizationResult[] = [];
    const combinations = this.generateParameterCombinations(parameterGrid);
    
    this.logger.info('Starting parameter optimization', { 
      strategy: strategyName,
      combinations: combinations.length 
    });
    
    // Use vectorized mode for fast parameter optimization
    const vectorMode = new VectorBacktestMode(config);
    
    // Can be parallelized
    await Promise.all(
      combinations.map(async (params) => {
        const strategy = await this.loadStrategy(strategyName, vectorMode, params);
        const result = await vectorMode.executeVectorizedBacktest(
          strategy as VectorizedStrategy
        );
        
        results.push({
          parameters: params,
          performance: result.performance,
          executionTime: result.executionTime
        });
      })
    );
    
    // Sort by Sharpe ratio
    return results.sort((a, b) => 
      b.performance.sharpeRatio - a.performance.sharpeRatio
    );
  }
}

Service Configuration

Environment-Based Mode Selection

export class ServiceConfig {
  getTradingConfig(): TradingConfig {
    return {
      mode: (process.env.TRADING_MODE as BacktestMode) || BacktestMode.LIVE,
      brokerConfig: {
        apiKey: process.env.BROKER_API_KEY,
        sandbox: process.env.BROKER_SANDBOX === 'true'
      },
      backtestConfig: {
        startDate: new Date(process.env.BACKTEST_START_DATE || '2023-01-01'),
        endDate: new Date(process.env.BACKTEST_END_DATE || '2024-01-01'),
        initialCapital: parseFloat(process.env.INITIAL_CAPITAL || '100000'),
        slippageModel: process.env.SLIPPAGE_MODEL || 'linear',
        commissionModel: process.env.COMMISSION_MODEL || 'fixed'
      }
    };
  }
}

CLI Interface

// CLI for running different modes
import { Command } from 'commander';

const program = new Command();

program
  .name('stock-bot')
  .description('Stock Trading Bot with Multi-Mode Backtesting');

program
  .command('live')
  .description('Run live trading')
  .option('-s, --strategy <strategy>', 'Strategy to run')
  .action(async (options) => {
    const strategyService = new StrategyService();
    await strategyService.runStrategy(
      options.strategy,
      BacktestMode.LIVE,
      {}
    );
  });

program
  .command('backtest')
  .description('Run backtesting')
  .option('-s, --strategy <strategy>', 'Strategy to test')
  .option('-m, --mode <mode>', 'Backtest mode (event|vector|hybrid)', 'event')
  .option('-f, --from <date>', 'Start date')
  .option('-t, --to <date>', 'End date')
  .action(async (options) => {
    const strategyService = new StrategyService();
    await strategyService.runStrategy(
      options.strategy,
      options.mode as BacktestMode,
      {
        startDate: new Date(options.from),
        endDate: new Date(options.to)
      }
    );
  });

program
  .command('optimize')
  .description('Optimize strategy parameters')
  .option('-s, --strategy <strategy>', 'Strategy to optimize')
  .option('-p, --params <params>', 'Parameter grid JSON')
  .action(async (options) => {
    const strategyService = new StrategyService();
    const paramGrid = JSON.parse(options.params);
    await strategyService.optimizeStrategy(
      options.strategy,
      paramGrid,
      {}
    );
  });

program.parse();

Performance Comparison

Execution Speed by Mode

Mode	Data Points/Second	Memory Usage	Use Case
Live	Real-time	Low	Production trading
Event-Driven	~1,000	Medium	Realistic validation
Vectorized	~100,000+	High	Parameter optimization
Hybrid	Combined	Medium	Strategy validation

When to Use Each Mode

Live Mode: Production trading with real money
Event-Driven: Final strategy validation, complex order logic
Vectorized: Initial development, parameter optimization, quick testing
Hybrid: Validating vectorized results against realistic simulation

Integration with Your Existing Libraries

This architecture leverages all your existing infrastructure:

@stock-bot/config: Environment management
@stock-bot/logger: Comprehensive logging with Loki
@stock-bot/utils: All technical indicators and calculations
@stock-bot/questdb-client: Time-series data storage
@stock-bot/postgres-client: Transactional data
@stock-bot/mongodb-client: Configuration storage

Key Benefits

Unified Codebase: Same strategy logic across all modes
Performance Flexibility: Choose speed vs accuracy based on needs
Validation Pipeline: Hybrid mode ensures vectorized results are accurate
Production Ready: Live mode for actual trading
Development Friendly: Fast iteration with vectorized backtesting

This simplified architecture reduces complexity while providing comprehensive backtesting capabilities that scale from rapid prototyping to production trading.

26 KiB Raw Blame History