initial setup

This commit is contained in:
Bojan Kucera 2025-06-02 08:15:20 -04:00
commit 232a63dfe8
61 changed files with 4985 additions and 0 deletions

233
docs/current-system-flow.md Normal file
View file

@ -0,0 +1,233 @@
# 🔄 Current System Communication Flow
## Active Services (Currently Running)
```mermaid
graph TB
%% External Data Sources
subgraph "External APIs"
EXT[Demo Data Generation<br/>Alpha Vantage Ready<br/>Yahoo Finance Ready]
end
%% Currently Active Services
subgraph "Active Services"
MDG[Market Data Gateway<br/>Port 3001<br/>✅ Running]
TD[Trading Dashboard<br/>Port 5173<br/>✅ Running]
end
%% Storage Layer
subgraph "Storage"
DRAGONFLY[(Dragonfly<br/>📡 Events & Cache<br/>🔧 Configured)]
end
%% Data Flow
EXT -->|HTTP Fetch| MDG
MDG -->|REST API| TD
MDG -->|WebSocket Stream| TD
MDG -->|Events| DRAGONFLY
%% Styling
classDef active fill:#90EE90
classDef storage fill:#FFE4B5
classDef external fill:#FFB6C1
class MDG,TD active
class DRAGONFLY storage
class EXT external
```
## Next Phase Services (Ready to Implement)
```mermaid
graph TB
%% Current Services
subgraph "Current Layer"
MDG[Market Data Gateway<br/>✅ Operational]
TD[Trading Dashboard<br/>✅ Operational]
end
%% Next Phase Services
subgraph "Next Phase - Priority 1"
SO[Strategy Orchestrator<br/>🚧 Package Created<br/>📋 Ready to Implement]
RG[Risk Guardian<br/>🚧 Package Created<br/>📋 Ready to Implement]
end
%% Communication Infrastructure
subgraph "Event Infrastructure"
DRAGONFLY[(Dragonfly Streams<br/>✅ Configured)]
WS[WebSocket Server<br/>✅ Active in MDG]
end
%% Data Flows
MDG -->|Market Data Events| DRAGONFLY
MDG -->|Real-time Stream| WS
WS -->|Live Updates| TD
DRAGONFLY -->|Market Events| SO
DRAGONFLY -->|All Events| RG
SO -->|Strategy Events| DRAGONFLY
SO -->|Risk Check| RG
RG -->|Risk Alerts| DRAGONFLY
%% Styling
classDef current fill:#90EE90
classDef next fill:#FFE4B5
classDef infrastructure fill:#E6E6FA
class MDG,TD current
class SO,RG next
class DRAGONFLY,WS infrastructure
```
## Detailed Communication Patterns
### 1. **Current System (Working)**
```
┌─────────────────┐ HTTP REST ┌─────────────────┐
│ Trading │ ←──────────────→ │ Market Data │
│ Dashboard │ │ Gateway │
│ (React/Tremor) │ ←──────────────→ │ (Hono/Bun) │
└─────────────────┘ WebSocket └─────────────────┘
┌─────────────────┐
│ Dragonfly Events │
│ (Configured) │
└─────────────────┘
```
### 2. **Next Phase Implementation**
```
┌─────────────────┐
│ Strategy │
│ Orchestrator │ ──┐
└─────────────────┘ │
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Market Data │→│ Dragonfly Event │←│ Risk Guardian │
│ Gateway │ │ Stream │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Trading │ │ WebSocket │ │ Alert Manager │
│ Dashboard │ │ Real-time │ │ (Future) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
```
## Event Flow Diagram
```mermaid
sequenceDiagram
participant TD as Trading Dashboard
participant MDG as Market Data Gateway
participant DRAGONFLY as Dragonfly Events
participant SO as Strategy Orchestrator (Next)
participant RG as Risk Guardian (Next)
Note over TD,DRAGONFLY: Current System - Working
MDG->>MDG: Generate demo market data
MDG->>TD: WebSocket real-time updates
MDG->>DRAGONFLY: Publish MARKET_DATA events
TD->>MDG: HTTP API requests
MDG->>TD: JSON responses
Note over SO,RG: Next Phase - To Implement
DRAGONFLY->>SO: Subscribe to MARKET_DATA events
SO->>SO: Analyze market conditions
SO->>DRAGONFLY: Publish SIGNAL_GENERATED events
DRAGONFLY->>RG: Subscribe to ALL events
RG->>RG: Monitor risk thresholds
RG->>DRAGONFLY: Publish RISK_ALERT events
DRAGONFLY->>TD: All events via WebSocket
TD->>TD: Update dashboard with alerts
```
## Service Dependencies
### **Current Dependencies (Satisfied)**
```
Market Data Gateway
├── ✅ Hono (Web framework)
├── ✅ ioredis (Redis client)
├── ✅ @stock-bot/config (Workspace package)
├── ✅ @stock-bot/shared-types (Workspace package)
└── ✅ ws (WebSocket library)
Trading Dashboard
├── ✅ React + TypeScript
├── ✅ Tremor UI (Financial components)
├── ✅ Vite (Build tool)
└── ✅ WebSocket client
```
### **Next Phase Dependencies (Ready)**
```
Strategy Orchestrator
├── ✅ Package.json created
├── ✅ Dependencies specified
├── 📋 Implementation needed
└── 🔧 node-cron (Strategy scheduling)
Risk Guardian
├── ✅ Package.json created
├── ✅ Dependencies specified
├── 📋 Implementation needed
└── 🛡️ Risk monitoring logic
```
## Port & Endpoint Map
| Service | Port | Endpoints | Status |
|---------|------|-----------|--------|
| **Market Data Gateway** | 3001 | `/health`, `/api/market-data/:symbol`, `/api/ohlcv/:symbol`, WebSocket | ✅ Active |
| **Trading Dashboard** | 5173 | Vite dev server | ✅ Active |
| **Strategy Orchestrator** | 4001 | `/health`, `/api/strategies`, `/api/signals` | 📋 Planned |
| **Risk Guardian** | 3002 | `/health`, `/api/risk-checks`, `/api/limits` | 📋 Planned |
## Data Types & Events
### **Market Data Event**
```typescript
interface MarketDataEvent {
type: 'MARKET_DATA';
data: {
symbol: string;
price: number;
bid: number;
ask: number;
volume: number;
timestamp: Date;
};
timestamp: Date;
}
```
### **Strategy Event (Next Phase)**
```typescript
interface StrategyEvent {
type: 'SIGNAL_GENERATED' | 'STRATEGY_START' | 'STRATEGY_STOP';
strategyId: string;
signal?: TradingSignal;
timestamp: Date;
}
```
### **Risk Event (Next Phase)**
```typescript
interface RiskEvent {
type: 'RISK_ALERT' | 'RISK_CHECK';
severity: 'LOW' | 'MEDIUM' | 'HIGH' | 'CRITICAL';
message: string;
data: any;
timestamp: Date;
}
```
This architecture shows how your current working system will expand into a comprehensive trading platform with clear communication patterns and event flows.

358
docs/system-architecture.md Normal file
View file

@ -0,0 +1,358 @@
# 🏗️ Stock Bot System Architecture
## System Communication Flow Diagram
```mermaid
graph TB
%% External Systems
subgraph "External APIs"
AV[Alpha Vantage API]
YF[Yahoo Finance API]
AL[Alpaca Broker API]
IB[Interactive Brokers]
NEWS[News APIs]
end
%% Core Services Layer
subgraph "Core Services"
MDG[Market Data Gateway<br/>:3001]
RG[Risk Guardian<br/>:3002]
EE[Execution Engine<br/>:3003]
PM[Portfolio Manager<br/>:3004]
end
%% Intelligence Services Layer
subgraph "Intelligence Services"
SO[Strategy Orchestrator<br/>:4001]
SG[Signal Generator<br/>:4002]
BA[Backtesting Engine<br/>:4003]
ML[ML Pipeline<br/>:4004]
end
%% Data Services Layer
subgraph "Data Services"
HDS[Historical Data Service<br/>:5001]
AS[Analytics Service<br/>:5002]
DQS[Data Quality Service<br/>:5003]
ETLS[ETL Service<br/>:5004]
end
%% Platform Services Layer
subgraph "Platform Services"
LM[Log Manager<br/>:6001]
CM[Config Manager<br/>:6002]
AM[Alert Manager<br/>:6003]
SM[Service Monitor<br/>:6004]
end
%% Integration Services Layer
subgraph "Integration Services"
BAS[Broker Adapter<br/>:7001]
DAS[Data Adapter<br/>:7002]
NS[Notification Service<br/>:7003]
WHS[Webhook Service<br/>:7004]
end
%% Interface Services Layer
subgraph "Interface Services"
TD[Trading Dashboard<br/>:5173]
API[REST API Gateway<br/>:8001]
WS[WebSocket Server<br/>Embedded]
end
%% Storage Layer
subgraph "Storage Layer"
DRAGONFLY[(Dragonfly<br/>Events & Cache)]
QDB[(QuestDB<br/>Time Series)]
PGDB[(PostgreSQL<br/>Relational)]
FS[(File System<br/>Logs & Config)]
end
%% Communication Flows
%% External to Core
AV --> MDG
YF --> MDG
AL --> BAS
IB --> BAS
NEWS --> DAS
%% Core Service Communications
MDG -->|Market Data Events| DRAGONFLY
MDG -->|Real-time Stream| WS
MDG -->|Cache| DRAGONFLY
RG -->|Risk Events| DRAGONFLY
RG -->|Risk Alerts| AM
RG -->|Position Limits| PM
EE -->|Order Events| DRAGONFLY
EE -->|Trade Execution| BAS
EE -->|Order Status| PM
PM -->|Portfolio Events| DRAGONFLY
PM -->|P&L Updates| TD
PM -->|Position Data| RG
%% Intelligence Communications
SO -->|Strategy Events| DRAGONFLY
SO -->|Signal Requests| SG
SO -->|Execution Orders| EE
SO -->|Risk Check| RG
SG -->|Trading Signals| SO
SG -->|ML Requests| ML
SG -->|Market Data| DRAGONFLY
BA -->|Backtest Results| SO
BA -->|Historical Data| HDS
ML -->|Predictions| SG
ML -->|Training Data| HDS
%% Data Service Communications
HDS -->|Store Data| QDB
HDS -->|Query Data| QDB
HDS -->|Data Events| DRAGONFLY
AS -->|Analytics| QDB
AS -->|Metrics| SM
AS -->|Reports| TD
DQS -->|Data Quality| DRAGONFLY
DQS -->|Alerts| AM
ETLS -->|Raw Data| DAS
ETLS -->|Processed Data| HDS
%% Platform Communications
LM -->|Logs| FS
LM -->|Log Events| DRAGONFLY
CM -->|Config| FS
CM -->|Config Updates| DRAGONFLY
AM -->|Alerts| NS
AM -->|Alert Events| DRAGONFLY
SM -->|Health Checks| DRAGONFLY
SM -->|Metrics| QDB
%% Integration Communications
BAS -->|Orders| AL
BAS -->|Orders| IB
BAS -->|Order Updates| EE
DAS -->|Data Feed| MDG
DAS -->|External Data| HDS
NS -->|Notifications| WHS
NS -->|Alerts| TD
WHS -->|Webhooks| External
%% Interface Communications
TD -->|API Calls| API
TD -->|WebSocket| WS
TD -->|Dashboard Data| PM
API -->|Service Calls| SO
API -->|Data Queries| HDS
API -->|System Status| SM
WS -->|Real-time Data| TD
WS -->|Events| DRAGONFLY
%% Storage Access DRAGONFLY -.->|Events| SO
DRAGONFLY -.->|Events| RG
DRAGONFLY -.->|Events| PM
DRAGONFLY -.->|Cache| MDG
QDB -.->|Time Series| HDS
QDB -.->|Analytics| AS
QDB -.->|Metrics| SM
PGDB -.->|Relational| PM
PGDB -.->|Config| CM
PGDB -.->|Users| API
%% Styling
classDef external fill:#ff9999
classDef core fill:#99ccff
classDef intelligence fill:#99ff99
classDef data fill:#ffcc99
classDef platform fill:#cc99ff
classDef integration fill:#ffff99
classDef interface fill:#ff99cc
classDef storage fill:#cccccc
class AV,YF,AL,IB,NEWS external
class MDG,RG,EE,PM core
class SO,SG,BA,ML intelligence
class HDS,AS,DQS,ETLS data
class LM,CM,AM,SM platform
class BAS,DAS,NS,WHS integration
class TD,API,WS interface
class DRAGONFLY,QDB,PGDB,FS storage
```
## Communication Patterns
### 1. **Event-Driven Architecture (Dragonfly Streams)**
```
┌─────────────┐ Dragonfly Events ┌─────────────┐
│ Service │ ─────────────────→ │ Service │
│ A │ │ B │
└─────────────┘ └─────────────┘
```
**Event Types:**
- `MARKET_DATA` - Real-time price updates
- `ORDER_CREATED/FILLED/CANCELLED` - Order lifecycle
- `SIGNAL_GENERATED` - Trading signals
- `RISK_ALERT` - Risk threshold violations
- `PORTFOLIO_UPDATE` - Position changes
- `STRATEGY_START/STOP` - Strategy lifecycle
### 2. **Request-Response (HTTP/REST)**
```
┌─────────────┐ HTTP Request ┌─────────────┐
│ Client │ ─────────────────→ │ Service │
│ │ ←───────────────── │ │
└─────────────┘ HTTP Response └─────────────┘
```
**API Endpoints:**
- `/api/market-data/:symbol` - Current market data
- `/api/portfolio/positions` - Portfolio positions
- `/api/strategies` - Strategy management
- `/api/orders` - Order management
- `/api/health` - Service health checks
### 3. **Real-time Streaming (WebSocket)**
```
┌─────────────┐ WebSocket ┌─────────────┐
│ Client │ ←═════════════════→ │ Server │
│ │ Bidirectional │ │
└─────────────┘ └─────────────┘
```
**WebSocket Messages:**
- Market data subscriptions
- Portfolio updates
- Trading signals
- Risk alerts
- System notifications
### 4. **Data Persistence**
```
┌─────────────┐ Store/Query ┌─────────────┐
│ Service │ ─────────────────→ │ Database │
│ │ ←───────────────── │ │
└─────────────┘ └─────────────┘
```
**Storage Types:**
- **Dragonfly**: Events, cache, sessions
- **QuestDB**: Time-series data, metrics
- **PostgreSQL**: Configuration, users, metadata
- **File System**: Logs, configurations
## Service Communication Matrix
| Service | Publishes Events | Subscribes to Events | HTTP APIs | WebSocket | Storage |
|---------|-----------------|---------------------|-----------|-----------|---------|
| Market Data Gateway | ✅ Market Data | - | ✅ REST | ✅ Server | Dragonfly Cache |
| Risk Guardian | ✅ Risk Alerts | ✅ All Events | ✅ REST | - | PostgreSQL |
| Strategy Orchestrator | ✅ Strategy Events | ✅ Market Data, Signals | ✅ REST | - | PostgreSQL |
| Execution Engine | ✅ Order Events | ✅ Strategy Events | ✅ REST | - | PostgreSQL |
| Portfolio Manager | ✅ Portfolio Events | ✅ Order Events | ✅ REST | - | PostgreSQL |
| Trading Dashboard | - | ✅ All Events | ✅ Client | ✅ Client | - |
## Data Flow Example: Trade Execution
```mermaid
sequenceDiagram
participant TD as Trading Dashboard
participant SO as Strategy Orchestrator
participant SG as Signal Generator
participant RG as Risk Guardian
participant EE as Execution Engine
participant BAS as Broker Adapter
participant PM as Portfolio Manager
participant DRAGONFLY as Dragonfly Events
Note over TD,DRAGONFLY: User starts a trading strategy
TD->>SO: POST /api/strategies/start
SO->>DRAGONFLY: Publish STRATEGY_START event
Note over SO,SG: Strategy generates signals
SO->>SG: Request signals for AAPL
SG->>SO: Return BUY signal
SO->>DRAGONFLY: Publish SIGNAL_GENERATED event
Note over SO,RG: Risk check before execution
SO->>RG: Check risk limits
RG->>SO: Risk approved
Note over SO,EE: Execute the trade
SO->>EE: Submit order
EE->>DRAGONFLY: Publish ORDER_CREATED event
EE->>BAS: Send order to broker
BAS->>EE: Order filled
EE->>DRAGONFLY: Publish ORDER_FILLED event
Note over PM,TD: Update portfolio and notify user
PM->>DRAGONFLY: Subscribe to ORDER_FILLED
PM->>PM: Update positions PM->>DRAGONFLY: Publish PORTFOLIO_UPDATE
TD->>DRAGONFLY: Subscribe to PORTFOLIO_UPDATE
TD->>TD: Update dashboard
```
## Port Allocation
| Service Category | Port Range | Services |
|-----------------|------------|----------|
| Core Services | 3001-3099 | Market Data Gateway (3001), Risk Guardian (3002) |
| Intelligence Services | 4001-4099 | Strategy Orchestrator (4001), Signal Generator (4002) |
| Data Services | 5001-5099 | Historical Data (5001), Analytics (5002) |
| Platform Services | 6001-6099 | Log Manager (6001), Config Manager (6002) |
| Integration Services | 7001-7099 | Broker Adapter (7001), Data Adapter (7002) |
| Interface Services | 8001-8099 | API Gateway (8001), Dashboard (5173-Vite) |
## Security & Authentication
```mermaid
graph LR
subgraph "Security Layer"
JWT[JWT Tokens]
API_KEY[API Keys]
TLS[TLS/HTTPS]
RBAC[Role-Based Access]
end
subgraph "External Security"
BROKER_AUTH[Broker Authentication]
DATA_AUTH[Data Provider Auth]
WEBHOOK_SIG[Webhook Signatures]
end
JWT --> API_KEY
API_KEY --> TLS
TLS --> RBAC
RBAC --> BROKER_AUTH
BROKER_AUTH --> DATA_AUTH
DATA_AUTH --> WEBHOOK_SIG
```
This architecture provides:
- **Scalability**: Services can be scaled independently
- **Reliability**: Event-driven communication with retry mechanisms
- **Maintainability**: Clear separation of concerns
- **Observability**: Centralized logging and monitoring
- **Security**: Multiple layers of authentication and authorization

View file

@ -0,0 +1,140 @@
# 📊 Stock Bot System Communication - Quick Reference
## Current System (Active)
```
┌─────────────────────────────────────────────────────────────────────┐
│ TRADING BOT SYSTEM │
└─────────────────────────────────────────────────────────────────────┘
External APIs Core Services Interface Services
┌─────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Demo Data │──────▶│ Market Data │◀──────▶│ Trading │
│ Alpha Vant. │ │ Gateway │ │ Dashboard │
│ Yahoo Fin. │ │ Port: 3001 │ │ Port: 5173 │
└─────────────┘ │ Status: ✅ LIVE │ │ Status: ✅ LIVE │
└─────────────────┘ └─────────────────┘
│ ▲
▼ │
┌─────────────────┐ │
│ Dragonfly Events │─────────────────┘
│ Cache & Streams │
│ Status: ✅ READY│
└─────────────────┘
```
## Next Phase (Ready to Implement)
```
┌─────────────────────────────────────────────────────────────────────┐
│ EXPANDED TRADING SYSTEM │
└─────────────────────────────────────────────────────────────────────┘
Intelligence Services Core Services Interface Services
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Strategy │◀────▶│ Market Data │◀───▶│ Trading │
│ Orchestrator │ │ Gateway │ │ Dashboard │
│ Port: 4001 │ │ Port: 3001 │ │ Port: 5173 │
│ Status: 📋 PLAN │ │ Status: ✅ LIVE │ │ Status: ✅ LIVE │
└─────────────────┘ └─────────────────┘ └─────────────────┘
▲ │ ▲
│ ▼ │
│ ┌─────────────────┐ │
└──────────────▶│ Dragonfly Event │◀─────────────┘
│ Stream Hub │
│ Status: ✅ READY│
└─────────────────┘
┌─────────────────┐
│ Risk Guardian │
│ Port: 3002 │
│ Status: 📋 PLAN │
└─────────────────┘
```
## Communication Protocols
### HTTP REST API
```
Client ──── GET/POST ───▶ Server
◀─── JSON ────────
```
### WebSocket Real-time
```
Client ◀═══ Stream ═══▶ Server
◀═══ Events ══▶
```
### Dragonfly Event Bus
```
Service A ──── Publish ───▶ Dragonfly ──── Subscribe ───▶ Service B
◀─── Confirm ──── ◀─── Events ────
```
## Event Types
| Event Type | Publisher | Subscribers | Frequency |
|------------|-----------|-------------|-----------|
| `MARKET_DATA` | Market Data Gateway | Dashboard, Strategy Orchestrator | Every 5s |
| `SIGNAL_GENERATED` | Strategy Orchestrator | Risk Guardian, Execution Engine | As needed |
| `RISK_ALERT` | Risk Guardian | Dashboard, Alert Manager | As needed |
| `PORTFOLIO_UPDATE` | Portfolio Manager | Dashboard, Risk Guardian | On trades |
## Service Status Matrix
| Service | Port | Status | Dependencies | Ready to Implement |
|---------|------|--------|--------------|-------------------|
| Market Data Gateway | 3001 | ✅ Running | Dragonfly, Config | ✅ Complete |
| Trading Dashboard | 5173 | ✅ Running | MDG API | ✅ Complete |
| Strategy Orchestrator | 4001 | 📋 Planned | Dragonfly, MDG | ✅ Package Ready |
| Risk Guardian | 3002 | 📋 Planned | Dragonfly, Config | ✅ Package Ready |
| Portfolio Manager | 3004 | ⏳ Future | Database, Orders | ❌ Not Started |
| Execution Engine | 3003 | ⏳ Future | Brokers, Portfolio | ❌ Not Started |
## Data Flow Summary
1. **Market Data Flow**
```
External APIs → Market Data Gateway → Dragonfly Events → Dashboard
→ Strategy Orchestrator
```
2. **Trading Signal Flow**
```
Market Data → Strategy Orchestrator → Trading Signals → Risk Guardian
→ Execution Engine
```
3. **Risk Management Flow**
```
All Events → Risk Guardian → Risk Alerts → Alert Manager
→ Risk Blocks → Strategy Orchestrator
```
4. **User Interface Flow**
```
WebSocket ← Dashboard → REST API → Services
Events ← → Commands →
```
## Implementation Priority
### Phase 1 (Current) ✅
- [x] Market Data Gateway
- [x] Trading Dashboard
- [x] Dragonfly Infrastructure
- [x] WebSocket Communication
### Phase 2 (Next) 📋
- [ ] Strategy Orchestrator
- [ ] Risk Guardian
- [ ] Event-driven Strategy Execution
- [ ] Risk Monitoring & Alerts
### Phase 3 (Future) ⏳
- [ ] Portfolio Manager
- [ ] Execution Engine
- [ ] Broker Integration
- [ ] Database Persistence
The system is designed for incremental development where each service can be implemented and tested independently while maintaining full system functionality.