diff --git a/DOCKER.md b/DOCKER.md deleted file mode 100644 index 2146043..0000000 --- a/DOCKER.md +++ /dev/null @@ -1,272 +0,0 @@ -# 🐳 Trading Bot Docker Infrastructure - -This directory contains the Docker Compose configuration for the Trading Bot infrastructure, including databases, caching, monitoring, and admin tools. - -## πŸ—οΈ Architecture Overview - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Trading Bot Infrastructure β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ πŸ‰ Dragonfly (Redis) β”‚ 🐘 PostgreSQL β”‚ πŸ“Š QuestDB β”‚ -β”‚ Events & Caching β”‚ Operational DB β”‚ Time-series Data β”‚ -β”‚ Port: 6379 β”‚ Port: 5432 β”‚ Port: 9000/8812 β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ πŸ”§ Redis Insight β”‚ πŸ› οΈ PgAdmin β”‚ πŸ“ˆ Prometheus β”‚ -β”‚ Dragonfly GUI β”‚ PostgreSQL GUI β”‚ Metrics Collection β”‚ -β”‚ Port: 8001 β”‚ Port: 8080 β”‚ Port: 9090 β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ πŸ“Š Grafana Dashboard β”‚ -β”‚ Visualization & Alerting β”‚ -β”‚ Port: 3000 β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -## πŸš€ Quick Start - -### 1. Start Core Infrastructure -```powershell -# Start essential services (Dragonfly, PostgreSQL, QuestDB) -npm run infra:up - -# Or use the management script -npm run docker:start -``` - -### 2. Start Admin Interfaces -```powershell -# Start Redis Insight and PgAdmin -npm run docker:admin -``` - -### 3. Start Full Development Environment -```powershell -# Start infrastructure + admin + your trading services -npm run dev:full -``` - -## πŸ“Š Service Access Points - -| Service | URL | Credentials | Purpose | -|---------|-----|-------------|---------| -| **Dragonfly** | `localhost:6379` | None | Redis-compatible cache & events | -| **PostgreSQL** | `localhost:5432` | `trading_user` / `trading_pass_dev` | Operational database | -| **QuestDB Console** | http://localhost:9000 | None | Time-series database GUI | -| **QuestDB Wire** | `localhost:8812` | `admin` / `quest` | PostgreSQL protocol access | -| **Redis Insight** | http://localhost:8001 | None | Dragonfly/Redis management | -| **PgAdmin** | http://localhost:8080 | `admin@tradingbot.local` / `admin123` | PostgreSQL management | -| **Prometheus** | http://localhost:9090 | None | Metrics collection | -| **Grafana** | http://localhost:3000 | `admin` / `admin123` | Dashboards & alerts | - -## πŸ› οΈ Management Commands - -### Using NPM Scripts (Recommended) -```powershell -# Infrastructure management -npm run docker:start # Start core services -npm run docker:stop # Stop all services -npm run docker:restart # Restart services -npm run docker:status # Show service status -npm run docker:logs # Show logs -npm run docker:reset # Reset all data (destructive) - -# Additional services -npm run docker:admin # Start admin interfaces -npm run docker:monitoring # Start monitoring stack - -# Quick development -npm run dev:full # Infrastructure + admin + services -npm run dev:clean # Reset + full restart -``` - -### Using PowerShell Script Directly -```powershell -# Basic operations -./scripts/docker.ps1 start -./scripts/docker.ps1 stop -./scripts/docker.ps1 status - -# View logs for specific service -./scripts/docker.ps1 logs -Service dragonfly - -# Development mode (lighter configuration) -./scripts/docker.ps1 start -Dev - -# Get help -./scripts/docker.ps1 help -``` - -### Using Docker Compose Directly -```powershell -# Start core services -docker-compose up -d dragonfly postgres questdb - -# Start with development overrides -docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d - -# Start monitoring (optional) -docker-compose --profile monitoring up -d - -# View logs -docker-compose logs -f - -# Stop everything -docker-compose down - -# Reset all data -docker-compose down -v -``` - -## πŸ—ƒοΈ Database Schemas - -### PostgreSQL Structure -```sql -trading.* -- Orders, positions, executions, accounts -strategy.* -- Strategies, signals, performance -risk.* -- Risk limits, events, monitoring -audit.* -- System events, health, configuration -``` - -### Initial Data -The PostgreSQL database is automatically initialized with: -- Common stock symbols (AAPL, GOOGL, etc.) -- Demo trading account with $100k -- Sample mean reversion strategy -- Basic risk limits -- Audit triggers for data tracking - -## πŸ“ˆ Monitoring & Observability - -### Prometheus Metrics -The setup includes Prometheus configuration to monitor: -- Trading bot services (ports 3001, 3002, 4001) -- Dragonfly performance -- PostgreSQL health -- QuestDB metrics - -### Grafana Dashboards -Access Grafana at http://localhost:3000 to view: -- Trading system performance -- Database metrics -- Service health -- Risk monitoring - -## πŸ”§ Configuration - -### Environment Variables -Services use these environment variables (configured in docker-compose.yml): - -```yaml -# Dragonfly -DRAGONFLY_HOST=localhost -DRAGONFLY_PORT=6379 - -# PostgreSQL -POSTGRES_HOST=localhost -POSTGRES_PORT=5432 -POSTGRES_DB=trading_bot -POSTGRES_USER=trading_user -POSTGRES_PASSWORD=trading_pass_dev - -# QuestDB -QUESTDB_HOST=localhost -QUESTDB_PORT=8812 -QUESTDB_DB=qdb -``` - -### Development vs Production -- **Development**: Use `docker-compose.dev.yml` for lighter resource usage -- **Production**: Use base `docker-compose.yml` with proper secrets management - -## 🚨 Troubleshooting - -### Common Issues - -**Port Conflicts** -```powershell -# Check what's using a port -netstat -ano | findstr :6379 - -# Kill process if needed -taskkill /PID /F -``` - -**Permission Issues** -```powershell -# Ensure Docker is running as administrator -# Check Docker Desktop settings -``` - -**Data Corruption** -```powershell -# Reset all data and restart fresh -npm run docker:reset -``` - -**Service Won't Start** -```powershell -# Check logs for specific service -npm run docker:logs -./scripts/docker.ps1 logs -Service dragonfly -``` - -### Health Checks -All services include health checks. Monitor with: -```powershell -npm run docker:status -docker ps --filter "name=trading-bot" -``` - -## πŸ”’ Security Notes - -### Development Security -- Default passwords are for development only -- No SSL/TLS in development mode -- Services exposed on localhost only - -### Production Considerations -- Change all default passwords -- Enable SSL/TLS encryption -- Use Docker secrets for sensitive data -- Implement proper backup strategies -- Set up log rotation -- Configure firewall rules - -## πŸ“‹ Maintenance - -### Backup Data -```powershell -# Backup PostgreSQL -docker exec trading-bot-postgres pg_dump -U trading_user trading_bot > backup.sql - -# Backup Dragonfly (if persistence enabled) -docker exec trading-bot-dragonfly redis-cli BGSAVE -``` - -### Update Services -```powershell -# Pull latest images -docker-compose pull - -# Restart with new images -npm run docker:restart -``` - -### Clean Up -```powershell -# Remove unused containers and images -docker system prune - -# Remove all trading-bot related containers and volumes -docker-compose down -v --remove-orphans -``` - -## 🎯 Next Steps - -1. **Start the infrastructure**: `npm run docker:start` -2. **Open admin interfaces**: `npm run docker:admin` -3. **Configure your services**: Update .env files to use Docker services -4. **Start development**: `npm run dev:full` -5. **Monitor health**: Check http://localhost:8001 and http://localhost:8080 - -For more advanced configuration, see the individual service documentation in the `database/` and `monitoring/` directories. diff --git a/SIMPLIFIED-ARCHITECTURE.md b/SIMPLIFIED-ARCHITECTURE.md index 9023561..bfaf979 100644 --- a/SIMPLIFIED-ARCHITECTURE.md +++ b/SIMPLIFIED-ARCHITECTURE.md @@ -1,54 +1,97 @@ -# Stock Trading Bot - Simplified Combined Architecture +# Stock Bot - System Architecture + +> **Updated**: June 2025 ## Overview -A TypeScript-based simplified microservices architecture for automated stock trading with integrated multi-mode backtesting capabilities. The system combines live trading and backtesting functionality into unified services while supporting live, event-driven, and vectorized backtesting modes. +TypeScript microservices architecture for automated stock trading with real-time data processing and multi-database storage. -## Architecture Diagram +## Core Services ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Data Service β”‚ β”‚Processing Serviceβ”‚ β”‚Strategy Service β”‚ -β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ -β”‚ β€’ Live APIs │────▢│ β€’ Indicators │────▢│ β€’ Live Mode β”‚ -β”‚ β€’ Historical β”‚ β”‚ β€’ Processing β”‚ β”‚ β€’ Event Mode β”‚ -β”‚ β€’ QuestDB β”‚ β”‚ β€’ Validation β”‚ β”‚ β€’ Vector Mode β”‚ -β”‚ β€’ Unified API β”‚ β”‚ β€’ Utils Lib β”‚ β”‚ β€’ Backtesting β”‚ +β”‚ β€’ Market Data │────▢│ β€’ Indicators │────▢│ β€’ Strategies β”‚ +β”‚ β€’ Providers β”‚ β”‚ β€’ Analytics β”‚ β”‚ β€’ Backtesting β”‚ +β”‚ β€’ QuestDB β”‚ β”‚ β€’ Validation β”‚ β”‚ β€’ Signal Gen β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ - β”‚ β”‚ Event Bus β”‚ β”‚ - └──────────────▢│ (Dragonfly) β”‚β—€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + └──────────────▢│ Event Bus β”‚β—€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ (Dragonfly) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚Execution Serviceβ”‚ β”‚Portfolio Serviceβ”‚ β”‚ Dashboard β”‚ -β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ -β”‚ β€’ Live Brokers β”‚ β”‚ β€’ Positions β”‚ β”‚ β€’ Angular UI β”‚ -β”‚ β€’ Simulation β”‚ β”‚ β€’ Risk Mgmt β”‚ β”‚ β€’ Real-time β”‚ -β”‚ β€’ Mode Switch β”‚ β”‚ β€’ P&L Tracking β”‚ β”‚ β€’ Backtest UI β”‚ -β”‚ β€’ Unified API β”‚ β”‚ β€’ Performance β”‚ β”‚ β€’ Reports β”‚ +β”‚ β€’ Order Mgmt β”‚ β”‚ β€’ Positions β”‚ β”‚ β€’ Angular UI β”‚ +β”‚ β€’ Risk Control β”‚ β”‚ β€’ Risk Mgmt β”‚ β”‚ β€’ Real-time β”‚ +β”‚ β€’ Execution β”‚ β”‚ β€’ Performance β”‚ β”‚ β€’ Analytics β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` -## Simplified Service Structure - -### 6 Core Services (Instead of 12) +## Services Structure ``` stock-bot/ β”œβ”€β”€ apps/ -β”‚ β”œβ”€β”€ data-service/ # Combined data ingestion & historical -β”‚ β”‚ β”œβ”€β”€ src/ -β”‚ β”‚ β”‚ β”œβ”€β”€ providers/ # Live & historical data providers -β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ live/ # Real-time APIs (WebSocket, REST) -β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ historical/ # Historical data (Yahoo, Alpha Vantage) -β”‚ β”‚ β”‚ β”‚ └── unified.ts # Unified data interface -β”‚ β”‚ β”‚ β”œβ”€β”€ storage/ # QuestDB time-series storage -β”‚ β”‚ β”‚ β”œβ”€β”€ services/ -β”‚ β”‚ β”‚ └── index.ts -β”‚ β”‚ └── package.json -β”‚ β”‚ +β”‚ β”œβ”€β”€ data-service/ # Market data ingestion & storage +β”‚ β”œβ”€β”€ execution-service/ # Order execution & broker integration +β”‚ β”œβ”€β”€ portfolio-service/ # Position & risk management +β”‚ β”œβ”€β”€ processing-service/ # Data processing & indicators +β”‚ β”œβ”€β”€ strategy-service/ # Trading strategies & backtesting +β”‚ └── dashboard/ # Angular UI (port 4200) +β”‚ +β”œβ”€β”€ libs/ # Shared libraries +β”‚ β”œβ”€β”€ logger/ # Centralized logging w/ Loki +β”‚ β”œβ”€β”€ config/ # Configuration management +β”‚ β”œβ”€β”€ event-bus/ # Event system +β”‚ β”œβ”€β”€ mongodb-client/ # MongoDB operations +β”‚ β”œβ”€β”€ postgres-client/ # PostgreSQL operations +β”‚ β”œβ”€β”€ questdb-client/ # Time-series data +β”‚ β”œβ”€β”€ http/ # HTTP client w/ proxy support +β”‚ β”œβ”€β”€ cache/ # Caching layer +β”‚ └── utils/ # Common utilities +β”‚ +└── database/ # Database configurations + β”œβ”€β”€ mongodb/init/ + └── postgres/init/ +``` + +## Technology Stack + +| Component | Technology | Purpose | +|-----------|------------|---------| +| **Runtime** | Bun | Fast JavaScript runtime | +| **Language** | TypeScript | Type-safe development | +| **Databases** | PostgreSQL, MongoDB, QuestDB | Multi-database architecture | +| **Caching** | Dragonfly (Redis) | Event bus & caching | +| **Frontend** | Angular 18 | Modern reactive UI | +| **Monitoring** | Prometheus, Grafana, Loki | Observability stack | + +## Quick Start + +```bash +# Install dependencies +bun install + +# Start infrastructure +bun run infra:up + +# Start services +bun run dev + +# Access dashboard +# http://localhost:4200 +``` + +## Key Features + +- **Real-time Trading**: Live market data & order execution +- **Multi-Database**: PostgreSQL, MongoDB, QuestDB for different data types +- **Event-Driven**: Asynchronous communication via Dragonfly +- **Monitoring**: Full observability with metrics, logs, and tracing +- **Modular**: Shared libraries for common functionality +- **Type-Safe**: Full TypeScript coverage β”‚ β”œβ”€β”€ processing-service/ # Combined processing & indicators β”‚ β”‚ β”œβ”€β”€ src/ β”‚ β”‚ β”‚ β”œβ”€β”€ indicators/ # Technical indicators (uses @stock-bot/utils) diff --git a/docs/MULTI_DATABASE_ARCHITECTURE.md b/docs/MULTI_DATABASE_ARCHITECTURE.md deleted file mode 100644 index 88d7071..0000000 --- a/docs/MULTI_DATABASE_ARCHITECTURE.md +++ /dev/null @@ -1,238 +0,0 @@ -# Stock Bot Multi-Database Architecture Documentation - -## Overview - -The Stock Bot platform uses a sophisticated multi-database architecture designed to handle different types of data efficiently. This document outlines the configuration system, database choices, and monitoring setup. - -## Configuration System - -### Migration from Custom Config to Envalid - -The platform has migrated from a complex Valibot-based configuration system to a simpler, more maintainable **envalid** approach: - -```typescript -// New configuration pattern used throughout -export const configName = cleanEnv(process.env, { - ENV_VAR: str({ default: 'value', desc: 'Description' }), - NUMERIC_VAR: num({ default: 3000, desc: 'Port number' }), - BOOLEAN_VAR: bool({ default: false, desc: 'Feature flag' }), -}); -``` - -### Configuration Modules - -| Module | Purpose | File | -|--------|---------|------| -| `database` | PostgreSQL operational data | `libs/config/src/database.ts` | -| `questdb` | Time-series data storage | `libs/config/src/questdb.ts` | -| `mongodb` | Document and unstructured data | `libs/config/src/mongodb.ts` | -| `dragonfly` | Caching and event streaming | `libs/config/src/dragonfly.ts` | -| `monitoring` | Prometheus and Grafana | `libs/config/src/monitoring.ts` | -| `loki` | Log aggregation | `libs/config/src/loki.ts` | -| `logging` | Application logging | `libs/config/src/logging.ts` | - -## Database Architecture - -### 1. PostgreSQL - Operational Data Store -**Purpose**: Primary relational database for structured operational data -- **Data Types**: Orders, positions, strategies, user accounts, trading rules -- **Strengths**: ACID compliance, complex queries, transactions -- **Configuration**: `libs/config/src/database.ts` - -```typescript -// Example usage -import { databaseConfig } from '@trading-bot/config'; -// Connects to operational PostgreSQL instance -``` - -### 2. QuestDB - Time-Series Database -**Purpose**: High-performance time-series data storage -- **Data Types**: OHLCV data, technical indicators, performance metrics, tick data -- **Strengths**: Fast ingestion, SQL queries on time-series, columnar storage -- **Configuration**: `libs/config/src/questdb.ts` - -```typescript -// Example usage -import { questdbConfig } from '@trading-bot/config'; -// Optimized for time-series queries and analytics -``` - -### 3. MongoDB - Document Store -**Purpose**: Flexible document storage for unstructured data -- **Data Types**: Market sentiment, news articles, research reports, ML model outputs -- **Strengths**: Schema flexibility, horizontal scaling, complex document queries -- **Configuration**: `libs/config/src/mongodb.ts` - -```typescript -// Example usage -import { mongodbConfig } from '@trading-bot/config'; -// Handles variable schema and complex nested data -``` - -### 4. Dragonfly - Cache & Event Store -**Purpose**: High-performance caching and real-time event streaming -- **Data Types**: Market data cache, session data, real-time events, pub/sub messages -- **Strengths**: Redis compatibility, better performance, memory efficiency -- **Configuration**: `libs/config/src/dragonfly.ts` - -```typescript -// Example usage -import { dragonflyConfig } from '@trading-bot/config'; -// Drop-in Redis replacement with better performance -``` - -## Monitoring & Observability Stack - -### Prometheus - Metrics Collection -- **Purpose**: Time-series metrics and monitoring -- **Metrics**: System performance, trading metrics, database metrics -- **Configuration**: `libs/config/src/monitoring.ts` - -### Grafana - Visualization -- **Purpose**: Dashboards and alerting -- **Dashboards**: Trading performance, system health, database monitoring -- **Configuration**: `libs/config/src/monitoring.ts` - -### Loki - Log Aggregation -- **Purpose**: Centralized log collection and analysis -- **Logs**: Application logs, database logs, system logs -- **Configuration**: `libs/config/src/loki.ts` - -### Application Logging -- **Purpose**: Structured application logging -- **Features**: Multiple formats, file rotation, log levels -- **Configuration**: `libs/config/src/logging.ts` - -## Environment Files - -### `.env` - Development (Local) -- **Purpose**: Local development with services on localhost -- **Databases**: All services running on localhost with standard ports -- **Logging**: Pretty-formatted console output - -### `.env.docker` - Docker Compose -- **Purpose**: Container orchestration with Docker Compose -- **Databases**: Container names as hostnames (e.g., `postgres`, `mongodb`) -- **Features**: Health checks, resource limits, volume mounts - -### `.env.complete` - Full Development -- **Purpose**: Complete feature set for development testing -- **Features**: All services enabled, verbose logging, debug mode -- **Use Case**: Testing the full platform locally - -### `.env.prod` - Production -- **Purpose**: Production deployment configuration -- **Security**: Environment variable references, secure defaults -- **Features**: Optimized logging, monitoring enabled - -## Admin Interfaces - -### PgAdmin - PostgreSQL Management -- **URL**: http://localhost:8080 (development) -- **Purpose**: Database administration, query execution, monitoring - -### Mongo Express - MongoDB Management -- **URL**: http://localhost:8081 (development) -- **Purpose**: Document browsing, collection management, query testing - -### Redis Insight - Dragonfly/Redis Management -- **URL**: http://localhost:8001 (development) -- **Purpose**: Cache monitoring, key browsing, performance analysis - -## Data Flow Architecture - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Market Data │───▢│ Dragonfly │───▢│ QuestDB β”‚ -β”‚ Feed β”‚ β”‚ (Cache/Events) β”‚ β”‚ (Time-Series) β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ - β–Ό -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Trading │◀──▢│ PostgreSQL β”‚ β”‚ MongoDB β”‚ -β”‚ Engine β”‚ β”‚ (Operational) β”‚ β”‚ (Documents) β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ β–² - β–Ό β”‚ -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ -β”‚ Monitoring │◀───│ Prometheus β”‚ β”‚ -β”‚ Dashboard β”‚ β”‚ (Metrics) β”‚ β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ - β”‚ -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ -β”‚ Log Analysis │◀───│ Loki β”‚β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -β”‚ β”‚ β”‚ (Logs) β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -## Best Practices - -### Database Selection Guidelines - -1. **PostgreSQL**: Use for transactional data requiring ACID properties - - Orders, positions, account balances, strategy configurations - -2. **QuestDB**: Use for time-series data requiring fast analytics - - OHLCV data, technical indicators, performance metrics - -3. **MongoDB**: Use for flexible, document-based data - - Market sentiment, news articles, ML model outputs - -4. **Dragonfly**: Use for temporary data requiring fast access - - Real-time market data cache, session data, event streams - -### Configuration Best Practices - -1. **Environment Separation**: Use appropriate `.env` file for each environment -2. **Security**: Never commit sensitive credentials to version control -3. **Validation**: All configuration uses envalid for runtime validation -4. **Documentation**: Each config variable includes descriptive help text - -### Monitoring Best Practices - -1. **Metrics**: Monitor database performance, trading metrics, system health -2. **Logging**: Use structured logging with appropriate log levels -3. **Alerting**: Set up Grafana alerts for critical system metrics -4. **Log Retention**: Configure appropriate retention periods for each environment - -## Migration Guide - -If migrating from the old configuration system: - -1. **Update Imports**: Change from custom config to new envalid-based modules -2. **Environment Variables**: Update `.env` files to include all new services -3. **Docker Setup**: Use `.env.docker` for container-based deployments -4. **Monitoring**: Enable Prometheus, Grafana, and Loki for observability - -## Troubleshooting - -### Common Issues - -1. **Connection Failures**: Check container names in Docker environments -2. **Port Conflicts**: Verify port mappings in environment files -3. **Permission Errors**: Ensure proper database credentials and permissions -4. **Memory Issues**: Adjust resource limits in Docker configuration - -### Debug Commands - -```bash -# Check container status -docker-compose ps - -# View container logs -docker-compose logs [service-name] - -# Test database connections -docker-compose exec postgres pg_isready -docker-compose exec mongodb mongosh --eval "db.runCommand('ping')" -docker-compose exec questdb curl -f http://localhost:9000/status -docker-compose exec dragonfly redis-cli ping -``` - -## Future Enhancements - -1. **Database Sharding**: Implement horizontal scaling for high-volume data -2. **Read Replicas**: Add read replicas for improved query performance -3. **Backup Strategy**: Implement automated backup and recovery procedures -4. **Security**: Add encryption at rest and in transit -5. **Performance**: Implement connection pooling and query optimization diff --git a/docs/bun-test-output-customization.md b/docs/bun-test-output-customization.md deleted file mode 100644 index 09a9d91..0000000 --- a/docs/bun-test-output-customization.md +++ /dev/null @@ -1,153 +0,0 @@ -# Bun Test Output Customization Guide - -## Command Line Options - -### Basic Output Control -```bash -# Run with coverage report -bun test --coverage - -# Use specific coverage reporter (text or lcov) -bun test --coverage --coverage-reporter=text -bun test --coverage --coverage-reporter=lcov -bun test --coverage --coverage-reporter=text,lcov - -# Set custom coverage directory -bun test --coverage --coverage-dir=my-coverage - -# Run only specific test name patterns -bun test --test-name-pattern="HttpClient" -bun test -t "should handle" - -# Stop after first failure -bun test --bail -bun test --bail=3 # Stop after 3 failures - -# Run only tests marked with .only() -bun test --only - -# Include todo tests -bun test --todo -``` - -### Reporters -```bash -# Use JUnit XML reporter for CI/CD -bun test --reporter=junit --reporter-outfile=test-results.xml - -# Default console output (no flag needed) -bun test -``` - -### Test Filtering -```bash -# Run specific test files -bun test http.test.ts -bun test "**/*integration*" - -# Run tests multiple times to catch flaky tests -bun test --rerun-each=3 -``` - -## Configuration in bunfig.toml - -You can set default options in your bunfig.toml file: - -```toml -[test] -# Always generate coverage -coverage = true - -# Set custom timeout -timeout = "10s" - -# Set default reporter -reporter = "junit" -reporter-outfile = "test-results.xml" - -# Set coverage options -coverage-reporter = "text,lcov" -coverage-dir = "coverage" - -# Test environment -preload = ["./test/setup.ts"] - -[test.env] -NODE_ENV = "test" -LOG_LEVEL = "silent" -``` - -## Custom Test Setup - -You can customize test output through your test setup files: - -### In test/setup.ts: -```typescript -// Customize console output -console.log = (message) => { - // Custom formatting - if (typeof message === 'string') { - console.info(`[TEST] ${new Date().toISOString()} ${message}`); - } -}; - -// Add custom test utilities -global.testHelpers = { - logTestStart: (testName: string) => { - console.info(`πŸ§ͺ Starting test: ${testName}`); - }, - logTestEnd: (testName: string, passed: boolean) => { - const icon = passed ? 'βœ…' : '❌'; - console.info(`${icon} Completed test: ${testName}`); - } -}; -``` - -### In individual test files: -```typescript -import { describe, test, expect, beforeEach } from 'bun:test'; - -describe('HttpClient', () => { - beforeEach(() => { - global.testHelpers?.logTestStart('HttpClient test'); - }); - - test('should work', () => { - // Your test code - global.testHelpers?.logTestEnd('should work', true); - }); -}); -``` - -## Package.json Scripts - -Add custom test scripts with different output configurations: - -```json -{ - "scripts": { - "test": "bun test", - "test:coverage": "bun test --coverage", - "test:junit": "bun test --reporter=junit --reporter-outfile=test-results.xml", - "test:watch": "bun test --watch", - "test:verbose": "bun test --coverage --coverage-reporter=text", - "test:ci": "bun test --coverage --reporter=junit --reporter-outfile=test-results.xml --bail", - "test:specific": "bun test --test-name-pattern" - } -} -``` - -## Environment Variables - -Control output through environment variables: - -```bash -# Set log level for tests -LOG_LEVEL=debug bun test - -# Silent mode -LOG_LEVEL=silent bun test - -# Custom formatting -TEST_OUTPUT_FORMAT=json bun test -``` diff --git a/docs/loki-logging.md b/docs/loki-logging.md index 8184bf0..fe8018f 100644 --- a/docs/loki-logging.md +++ b/docs/loki-logging.md @@ -1,18 +1,16 @@ # Loki Logging for Stock Bot -This document outlines how to use the Loki logging system integrated with the Stock Bot platform. +This document outlines how to use the Loki logging system integrated with the Stock Bot platform (Updated June 2025). ## Overview -Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs (labels), not the full text. This makes Loki more resource-efficient than traditional log storage systems. +Loki provides centralized logging for all Stock Bot services with: -For Stock Bot, Loki provides: - -1. Centralized logging for all services -2. Log aggregation and filtering by service, level, and custom labels -3. Integration with Grafana for visualization -4. Query capabilities for log analysis -5. Alert capabilities for critical issues +1. **Centralized logging** for all microservices +2. **Log aggregation** and filtering by service, level, and custom labels +3. **Grafana integration** for visualization and dashboards +4. **Query capabilities** using LogQL for log analysis +5. **Alert capabilities** for critical issues ## Getting Started @@ -23,7 +21,7 @@ For Stock Bot, Loki provides: scripts\docker.ps1 monitoring ``` -You can also start Loki directly using Docker Compose: +Or start services individually: ```cmd # Start Loki service only @@ -43,21 +41,21 @@ Once started: ## Using the Logger in Your Services -The Stock Bot logger has been enhanced to automatically send logs to Loki. Here's how to use it: +The Stock Bot logger automatically sends logs to Loki using the updated pattern: ```typescript -import { Logger, LogLevel } from '@stock-bot/utils'; +import { getLogger } from '@stock-bot/logger'; // Create a logger for your service -const logger = new Logger('your-service-name', LogLevel.INFO); +const logger = getLogger('your-service-name'); // Log at different levels logger.debug('Detailed information for debugging'); logger.info('General information about operations'); -logger.warn('Potential issues that don't affect operation'); +logger.warn('Potential issues that don\'t affect operation'); logger.error('Critical errors that require attention'); -// Log with structured data (will be searchable in Loki) +// Log with structured data (searchable in Loki) logger.info('Processing trade', { symbol: 'MSFT', price: 410.75, @@ -107,10 +105,10 @@ Inside Grafana, you can use these LogQL queries to analyze your logs: ## Testing the Logging Integration -A test script is provided to verify the logging integration: +Test the logging integration using Bun: -```bash -# Run from project root +```cmd +# Run from project root using Bun (current runtime) bun run tools/test-loki-logging.ts ``` @@ -120,8 +118,8 @@ Our logging implementation follows this architecture: ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Trading Services│────►│ @stock-bot/utilsβ”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Logger β”‚ +β”‚ Trading Services│────►│ @stock-bot/loggerβ”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ getLogger() β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β–Ό @@ -163,9 +161,9 @@ If logs aren't appearing in Grafana: type .env | findstr "LOKI_" ``` -4. Ensure your service has the latest @stock-bot/utils package +4. Ensure your service has the latest @stock-bot/logger package 5. Check for errors in the Loki container logs: ```cmd - docker logs trading-bot-loki + docker logs stock-bot-loki ``` diff --git a/docs/testing-with-bun.md b/docs/testing-with-bun.md index 5c1dc8c..586db8f 100644 --- a/docs/testing-with-bun.md +++ b/docs/testing-with-bun.md @@ -1,12 +1,12 @@ # Testing with Bun in Stock Bot Platform -This project uses [Bun Test](https://bun.sh/docs/cli/test) for all testing needs. Bun Test provides a fast, modern testing experience with Jest-like API compatibility. +The Stock Bot platform uses [Bun Test](https://bun.sh/docs/cli/test) as the primary testing framework (Updated June 2025). Bun Test provides fast, modern testing with Jest-like API compatibility. ## Getting Started -To run tests: +Run tests using these commands: -```bash +```cmd # Run all tests (using Turbo) bun test diff --git a/libs/logger/README.md b/libs/logger/README.md index 24a5f2a..dcf996a 100644 --- a/libs/logger/README.md +++ b/libs/logger/README.md @@ -1,6 +1,6 @@ # @stock-bot/logger -Enhanced logging library with Loki integration for the Stock Bot platform. +Enhanced logging library with Loki integration for the Stock Bot platform (June 2025). ## Features @@ -17,8 +17,8 @@ Enhanced logging library with Loki integration for the Stock Bot platform. ## Installation ```bash -cd libs/logger -npm install +# Using Bun (current runtime) +bun install ``` ## Basic Usage @@ -38,17 +38,16 @@ logger.error('An error occurred', new Error('Something went wrong')); ### With Context ```typescript -import { Logger } from '@stock-bot/logger'; +import { getLogger } from '@stock-bot/logger'; -const logger = new Logger('trading-service', { - userId: '12345', - sessionId: 'abc-def-ghi' -}); +const logger = getLogger('trading-service'); logger.info('Trade executed', { symbol: 'AAPL', quantity: 100, - price: 150.25 + price: 150.25, + userId: '12345', + sessionId: 'abc-def-ghi' }); ``` @@ -170,6 +169,8 @@ LOKI_BATCH_SIZE=1024 ### Child Loggers ```typescript +import { getLogger } from '@stock-bot/logger'; + const parentLogger = getLogger('trading-service'); const orderLogger = parentLogger.child({ module: 'order-processing', @@ -179,17 +180,13 @@ const orderLogger = parentLogger.child({ orderLogger.info('Order validated'); // Will include parent context ``` -### Custom Transports +### Custom Configuration ```typescript -import { createLogger } from '@stock-bot/logger'; +import { getLogger } from '@stock-bot/logger'; -const logger = createLogger('custom-service', { - level: 'debug', - enableLoki: true, - enableFile: false, - enableConsole: true -}); +// Uses standard getLogger with service-specific configuration +const logger = getLogger('custom-service'); ``` ### Sensitive Data Masking