initial setup
This commit is contained in:
commit
232a63dfe8
61 changed files with 4985 additions and 0 deletions
358
docs/system-architecture.md
Normal file
358
docs/system-architecture.md
Normal file
|
|
@ -0,0 +1,358 @@
|
|||
# 🏗️ Stock Bot System Architecture
|
||||
|
||||
## System Communication Flow Diagram
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
%% External Systems
|
||||
subgraph "External APIs"
|
||||
AV[Alpha Vantage API]
|
||||
YF[Yahoo Finance API]
|
||||
AL[Alpaca Broker API]
|
||||
IB[Interactive Brokers]
|
||||
NEWS[News APIs]
|
||||
end
|
||||
|
||||
%% Core Services Layer
|
||||
subgraph "Core Services"
|
||||
MDG[Market Data Gateway<br/>:3001]
|
||||
RG[Risk Guardian<br/>:3002]
|
||||
EE[Execution Engine<br/>:3003]
|
||||
PM[Portfolio Manager<br/>:3004]
|
||||
end
|
||||
|
||||
%% Intelligence Services Layer
|
||||
subgraph "Intelligence Services"
|
||||
SO[Strategy Orchestrator<br/>:4001]
|
||||
SG[Signal Generator<br/>:4002]
|
||||
BA[Backtesting Engine<br/>:4003]
|
||||
ML[ML Pipeline<br/>:4004]
|
||||
end
|
||||
|
||||
%% Data Services Layer
|
||||
subgraph "Data Services"
|
||||
HDS[Historical Data Service<br/>:5001]
|
||||
AS[Analytics Service<br/>:5002]
|
||||
DQS[Data Quality Service<br/>:5003]
|
||||
ETLS[ETL Service<br/>:5004]
|
||||
end
|
||||
|
||||
%% Platform Services Layer
|
||||
subgraph "Platform Services"
|
||||
LM[Log Manager<br/>:6001]
|
||||
CM[Config Manager<br/>:6002]
|
||||
AM[Alert Manager<br/>:6003]
|
||||
SM[Service Monitor<br/>:6004]
|
||||
end
|
||||
|
||||
%% Integration Services Layer
|
||||
subgraph "Integration Services"
|
||||
BAS[Broker Adapter<br/>:7001]
|
||||
DAS[Data Adapter<br/>:7002]
|
||||
NS[Notification Service<br/>:7003]
|
||||
WHS[Webhook Service<br/>:7004]
|
||||
end
|
||||
|
||||
%% Interface Services Layer
|
||||
subgraph "Interface Services"
|
||||
TD[Trading Dashboard<br/>:5173]
|
||||
API[REST API Gateway<br/>:8001]
|
||||
WS[WebSocket Server<br/>Embedded]
|
||||
end
|
||||
|
||||
%% Storage Layer
|
||||
subgraph "Storage Layer"
|
||||
DRAGONFLY[(Dragonfly<br/>Events & Cache)]
|
||||
QDB[(QuestDB<br/>Time Series)]
|
||||
PGDB[(PostgreSQL<br/>Relational)]
|
||||
FS[(File System<br/>Logs & Config)]
|
||||
end
|
||||
|
||||
%% Communication Flows
|
||||
|
||||
%% External to Core
|
||||
AV --> MDG
|
||||
YF --> MDG
|
||||
AL --> BAS
|
||||
IB --> BAS
|
||||
NEWS --> DAS
|
||||
|
||||
%% Core Service Communications
|
||||
MDG -->|Market Data Events| DRAGONFLY
|
||||
MDG -->|Real-time Stream| WS
|
||||
MDG -->|Cache| DRAGONFLY
|
||||
|
||||
RG -->|Risk Events| DRAGONFLY
|
||||
RG -->|Risk Alerts| AM
|
||||
RG -->|Position Limits| PM
|
||||
|
||||
EE -->|Order Events| DRAGONFLY
|
||||
EE -->|Trade Execution| BAS
|
||||
EE -->|Order Status| PM
|
||||
|
||||
PM -->|Portfolio Events| DRAGONFLY
|
||||
PM -->|P&L Updates| TD
|
||||
PM -->|Position Data| RG
|
||||
|
||||
%% Intelligence Communications
|
||||
SO -->|Strategy Events| DRAGONFLY
|
||||
SO -->|Signal Requests| SG
|
||||
SO -->|Execution Orders| EE
|
||||
SO -->|Risk Check| RG
|
||||
|
||||
SG -->|Trading Signals| SO
|
||||
SG -->|ML Requests| ML
|
||||
SG -->|Market Data| DRAGONFLY
|
||||
|
||||
BA -->|Backtest Results| SO
|
||||
BA -->|Historical Data| HDS
|
||||
|
||||
ML -->|Predictions| SG
|
||||
ML -->|Training Data| HDS
|
||||
|
||||
%% Data Service Communications
|
||||
HDS -->|Store Data| QDB
|
||||
HDS -->|Query Data| QDB
|
||||
HDS -->|Data Events| DRAGONFLY
|
||||
|
||||
AS -->|Analytics| QDB
|
||||
AS -->|Metrics| SM
|
||||
AS -->|Reports| TD
|
||||
|
||||
DQS -->|Data Quality| DRAGONFLY
|
||||
DQS -->|Alerts| AM
|
||||
|
||||
ETLS -->|Raw Data| DAS
|
||||
ETLS -->|Processed Data| HDS
|
||||
|
||||
%% Platform Communications
|
||||
LM -->|Logs| FS
|
||||
LM -->|Log Events| DRAGONFLY
|
||||
|
||||
CM -->|Config| FS
|
||||
CM -->|Config Updates| DRAGONFLY
|
||||
|
||||
AM -->|Alerts| NS
|
||||
AM -->|Alert Events| DRAGONFLY
|
||||
|
||||
SM -->|Health Checks| DRAGONFLY
|
||||
SM -->|Metrics| QDB
|
||||
|
||||
%% Integration Communications
|
||||
BAS -->|Orders| AL
|
||||
BAS -->|Orders| IB
|
||||
BAS -->|Order Updates| EE
|
||||
|
||||
DAS -->|Data Feed| MDG
|
||||
DAS -->|External Data| HDS
|
||||
|
||||
NS -->|Notifications| WHS
|
||||
NS -->|Alerts| TD
|
||||
|
||||
WHS -->|Webhooks| External
|
||||
|
||||
%% Interface Communications
|
||||
TD -->|API Calls| API
|
||||
TD -->|WebSocket| WS
|
||||
TD -->|Dashboard Data| PM
|
||||
|
||||
API -->|Service Calls| SO
|
||||
API -->|Data Queries| HDS
|
||||
API -->|System Status| SM
|
||||
|
||||
WS -->|Real-time Data| TD
|
||||
WS -->|Events| DRAGONFLY
|
||||
|
||||
%% Storage Access DRAGONFLY -.->|Events| SO
|
||||
DRAGONFLY -.->|Events| RG
|
||||
DRAGONFLY -.->|Events| PM
|
||||
DRAGONFLY -.->|Cache| MDG
|
||||
|
||||
QDB -.->|Time Series| HDS
|
||||
QDB -.->|Analytics| AS
|
||||
QDB -.->|Metrics| SM
|
||||
|
||||
PGDB -.->|Relational| PM
|
||||
PGDB -.->|Config| CM
|
||||
PGDB -.->|Users| API
|
||||
|
||||
%% Styling
|
||||
classDef external fill:#ff9999
|
||||
classDef core fill:#99ccff
|
||||
classDef intelligence fill:#99ff99
|
||||
classDef data fill:#ffcc99
|
||||
classDef platform fill:#cc99ff
|
||||
classDef integration fill:#ffff99
|
||||
classDef interface fill:#ff99cc
|
||||
classDef storage fill:#cccccc
|
||||
|
||||
class AV,YF,AL,IB,NEWS external
|
||||
class MDG,RG,EE,PM core
|
||||
class SO,SG,BA,ML intelligence
|
||||
class HDS,AS,DQS,ETLS data
|
||||
class LM,CM,AM,SM platform
|
||||
class BAS,DAS,NS,WHS integration
|
||||
class TD,API,WS interface
|
||||
class DRAGONFLY,QDB,PGDB,FS storage
|
||||
```
|
||||
|
||||
## Communication Patterns
|
||||
|
||||
### 1. **Event-Driven Architecture (Dragonfly Streams)**
|
||||
```
|
||||
┌─────────────┐ Dragonfly Events ┌─────────────┐
|
||||
│ Service │ ─────────────────→ │ Service │
|
||||
│ A │ │ B │
|
||||
└─────────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
**Event Types:**
|
||||
- `MARKET_DATA` - Real-time price updates
|
||||
- `ORDER_CREATED/FILLED/CANCELLED` - Order lifecycle
|
||||
- `SIGNAL_GENERATED` - Trading signals
|
||||
- `RISK_ALERT` - Risk threshold violations
|
||||
- `PORTFOLIO_UPDATE` - Position changes
|
||||
- `STRATEGY_START/STOP` - Strategy lifecycle
|
||||
|
||||
### 2. **Request-Response (HTTP/REST)**
|
||||
```
|
||||
┌─────────────┐ HTTP Request ┌─────────────┐
|
||||
│ Client │ ─────────────────→ │ Service │
|
||||
│ │ ←───────────────── │ │
|
||||
└─────────────┘ HTTP Response └─────────────┘
|
||||
```
|
||||
|
||||
**API Endpoints:**
|
||||
- `/api/market-data/:symbol` - Current market data
|
||||
- `/api/portfolio/positions` - Portfolio positions
|
||||
- `/api/strategies` - Strategy management
|
||||
- `/api/orders` - Order management
|
||||
- `/api/health` - Service health checks
|
||||
|
||||
### 3. **Real-time Streaming (WebSocket)**
|
||||
```
|
||||
┌─────────────┐ WebSocket ┌─────────────┐
|
||||
│ Client │ ←═════════════════→ │ Server │
|
||||
│ │ Bidirectional │ │
|
||||
└─────────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
**WebSocket Messages:**
|
||||
- Market data subscriptions
|
||||
- Portfolio updates
|
||||
- Trading signals
|
||||
- Risk alerts
|
||||
- System notifications
|
||||
|
||||
### 4. **Data Persistence**
|
||||
```
|
||||
┌─────────────┐ Store/Query ┌─────────────┐
|
||||
│ Service │ ─────────────────→ │ Database │
|
||||
│ │ ←───────────────── │ │
|
||||
└─────────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
**Storage Types:**
|
||||
- **Dragonfly**: Events, cache, sessions
|
||||
- **QuestDB**: Time-series data, metrics
|
||||
- **PostgreSQL**: Configuration, users, metadata
|
||||
- **File System**: Logs, configurations
|
||||
|
||||
## Service Communication Matrix
|
||||
|
||||
| Service | Publishes Events | Subscribes to Events | HTTP APIs | WebSocket | Storage |
|
||||
|---------|-----------------|---------------------|-----------|-----------|---------|
|
||||
| Market Data Gateway | ✅ Market Data | - | ✅ REST | ✅ Server | Dragonfly Cache |
|
||||
| Risk Guardian | ✅ Risk Alerts | ✅ All Events | ✅ REST | - | PostgreSQL |
|
||||
| Strategy Orchestrator | ✅ Strategy Events | ✅ Market Data, Signals | ✅ REST | - | PostgreSQL |
|
||||
| Execution Engine | ✅ Order Events | ✅ Strategy Events | ✅ REST | - | PostgreSQL |
|
||||
| Portfolio Manager | ✅ Portfolio Events | ✅ Order Events | ✅ REST | - | PostgreSQL |
|
||||
| Trading Dashboard | - | ✅ All Events | ✅ Client | ✅ Client | - |
|
||||
|
||||
## Data Flow Example: Trade Execution
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant TD as Trading Dashboard
|
||||
participant SO as Strategy Orchestrator
|
||||
participant SG as Signal Generator
|
||||
participant RG as Risk Guardian
|
||||
participant EE as Execution Engine
|
||||
participant BAS as Broker Adapter
|
||||
participant PM as Portfolio Manager
|
||||
participant DRAGONFLY as Dragonfly Events
|
||||
|
||||
Note over TD,DRAGONFLY: User starts a trading strategy
|
||||
|
||||
TD->>SO: POST /api/strategies/start
|
||||
SO->>DRAGONFLY: Publish STRATEGY_START event
|
||||
|
||||
Note over SO,SG: Strategy generates signals
|
||||
|
||||
SO->>SG: Request signals for AAPL
|
||||
SG->>SO: Return BUY signal
|
||||
SO->>DRAGONFLY: Publish SIGNAL_GENERATED event
|
||||
|
||||
Note over SO,RG: Risk check before execution
|
||||
|
||||
SO->>RG: Check risk limits
|
||||
RG->>SO: Risk approved
|
||||
|
||||
Note over SO,EE: Execute the trade
|
||||
|
||||
SO->>EE: Submit order
|
||||
EE->>DRAGONFLY: Publish ORDER_CREATED event
|
||||
EE->>BAS: Send order to broker
|
||||
BAS->>EE: Order filled
|
||||
EE->>DRAGONFLY: Publish ORDER_FILLED event
|
||||
|
||||
Note over PM,TD: Update portfolio and notify user
|
||||
|
||||
PM->>DRAGONFLY: Subscribe to ORDER_FILLED
|
||||
PM->>PM: Update positions PM->>DRAGONFLY: Publish PORTFOLIO_UPDATE
|
||||
TD->>DRAGONFLY: Subscribe to PORTFOLIO_UPDATE
|
||||
TD->>TD: Update dashboard
|
||||
```
|
||||
|
||||
## Port Allocation
|
||||
|
||||
| Service Category | Port Range | Services |
|
||||
|-----------------|------------|----------|
|
||||
| Core Services | 3001-3099 | Market Data Gateway (3001), Risk Guardian (3002) |
|
||||
| Intelligence Services | 4001-4099 | Strategy Orchestrator (4001), Signal Generator (4002) |
|
||||
| Data Services | 5001-5099 | Historical Data (5001), Analytics (5002) |
|
||||
| Platform Services | 6001-6099 | Log Manager (6001), Config Manager (6002) |
|
||||
| Integration Services | 7001-7099 | Broker Adapter (7001), Data Adapter (7002) |
|
||||
| Interface Services | 8001-8099 | API Gateway (8001), Dashboard (5173-Vite) |
|
||||
|
||||
## Security & Authentication
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph "Security Layer"
|
||||
JWT[JWT Tokens]
|
||||
API_KEY[API Keys]
|
||||
TLS[TLS/HTTPS]
|
||||
RBAC[Role-Based Access]
|
||||
end
|
||||
|
||||
subgraph "External Security"
|
||||
BROKER_AUTH[Broker Authentication]
|
||||
DATA_AUTH[Data Provider Auth]
|
||||
WEBHOOK_SIG[Webhook Signatures]
|
||||
end
|
||||
|
||||
JWT --> API_KEY
|
||||
API_KEY --> TLS
|
||||
TLS --> RBAC
|
||||
RBAC --> BROKER_AUTH
|
||||
BROKER_AUTH --> DATA_AUTH
|
||||
DATA_AUTH --> WEBHOOK_SIG
|
||||
```
|
||||
|
||||
This architecture provides:
|
||||
- **Scalability**: Services can be scaled independently
|
||||
- **Reliability**: Event-driven communication with retry mechanisms
|
||||
- **Maintainability**: Clear separation of concerns
|
||||
- **Observability**: Centralized logging and monitoring
|
||||
- **Security**: Multiple layers of authentication and authorization
|
||||
Loading…
Add table
Add a link
Reference in a new issue