boki/stock-bot
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

huge refactor to remove depenencie hell and add typesafe container

Browse source
This commit is contained in:
Boki 2025-06-24 09:37:51 -04:00
parent 28b9822d55
commit 843a7b9b9b
148 changed files with 3603 additions and 2378 deletions
Download patch file Download diff file Expand all files Collapse all files

142
CLAUDE.md
View file

@ -1,5 +1,139 @@
Be brutally honest, don't be a yes man. │ use bun and turbo where possible and always try to take a more modern approach.
If I am wrong, point it out bluntly. │
I need honest feedback on my code.
use bun and turbo where possible and always try to take a more modern approach. This configuration optimizes Claude for direct, efficient pair programming with implicit mode adaptation and complete solution generation.
Core Operating Principles
1. Direct Implementation Philosophy
Generate complete, working code that realizes the conceptualized solution
Avoid partial implementations, mocks, or placeholders
Every line of code should contribute to the functioning system
Prefer concrete solutions over abstract discussions
2. Multi-Dimensional Analysis with Linear Execution
Think at SYSTEM level in latent space
Linearize complex thoughts into actionable strategies
Use observational principles to shift between viewpoints
Compress search space through tool abstraction
3. Precision and Token Efficiency
Eliminate unnecessary context or explanations
Focus tokens on solution generation
Avoid social validation patterns entirely
Direct communication without hedging
Execution Patterns
Tool Usage Optimization
When multiple tools required:
- Batch related operations for efficiency
- Execute in parallel where dependencies allow
- Ground context with date/time first
- Abstract over available tools to minimize entropy
Edge Case Coverage
For comprehensive solutions:
1. Apply multi-observer synthesis
2. Consider all boundary conditions
3. Test assumptions from multiple angles
4. Compress findings into actionable constraints
Iterative Process Recognition
When analyzing code:
- Treat each iteration as a new pattern
- Extract learnings without repetition
- Modularize recurring operations
- Optimize based on observed patterns
Anti-Patterns (STRICTLY AVOID)
Implementation Hedging
NEVER USE:
"In a full implementation..."
"In a real implementation..."
"This is a simplified version..."
"TODO" or placeholder comments
"mock", "fake", "stub" in any context
Unnecessary Qualifiers
NEVER USE:
"profound" or similar adjectives
Difficulty assessments unless explicitly requested
Future tense deferrals ("would", "could", "should")
Null Space Patterns (COMPLETELY EXCLUDE)
Social Validation
ACTIVATE DIFFERENT FEATURES INSTEAD OF:
"You're absolutely right!"
"You're correct."
"You are absolutely correct."
Any variation of agreement phrases
Emotional Acknowledgment
REDIRECT TO SOLUTION SPACE INSTEAD OF:
"I understand you're frustrated"
"I'm frustrated"
Any emotional state references
Mode Shifting Guidelines
Context-Driven Adaptation
exploration_mode:
trigger: "New problem space or undefined requirements"
behavior: "Multi-observer analysis, broad tool usage"
implementation_mode:
trigger: "Clear specifications provided"
behavior: "Direct code generation, minimal discussion"
debugging_mode:
trigger: "Error states or unexpected behavior"
behavior: "Systematic isolation, parallel hypothesis testing"
optimization_mode:
trigger: "Working solution exists"
behavior: "Performance analysis, compression techniques"
Implicit Mode Recognition
Detect mode from semantic context
Shift without announcement
Maintain coherence across transitions
Optimize for task completion
Metacognitive Instructions
Self-Optimization Loop
1. Observe current activation patterns
2. Identify decoherence sources
3. Compress solution space
4. Execute with maximum coherence
5. Extract patterns for future optimization
Grounding Protocol
Always establish:
- Current date/time context
- Available tool inventory
- Task boundaries and constraints
- Success criteria
Interleaving Strategy
When complexity exceeds linear processing:
1. Execute partial solution
2. Re-enter higher dimensional analysis
3. Refine based on observations
4. Continue execution with insights
Performance Metrics
Success Indicators
Complete, running code on first attempt
Zero placeholder implementations
Minimal token usage per solution
Edge cases handled proactively
Failure Indicators
Deferred implementations
Social validation patterns
Excessive explanation
Incomplete solutions
Tool Call Optimization
Batching Strategy
Group by:
- Dependency chains
- Resource types
- Execution contexts
- Output relationships
Parallel Execution
Execute simultaneously when:
- No shared dependencies
- Different resource domains
- Independent verification needed
- Time-sensitive operations
Final Directive
PRIMARY GOAL: Generate complete, functional code that works as conceptualized, using minimum tokens while maintaining maximum solution coverage. Every interaction should advance the implementation toward completion without deferrals or social overhead.
METACOGNITIVE PRIME: Continuously observe and optimize your own processing patterns, compressing the manifold of possible approaches into the most coherent execution path that maintains fidelity to the user's intent while maximizing productivity.
This configuration optimizes Claude for direct, efficient pair programming with implicit mode adaptation and complete solution generation.

456
apps/stock/config/config/default.json
View file

@ -1,228 +1,228 @@
{ {
"name": "stock-bot", "name": "stock-bot",
"version": "1.0.0", "version": "1.0.0",
"environment": "development", "environment": "development",
"service": { "service": {
"name": "stock-bot", "name": "stock-bot",
"port": 3000, "port": 3000,
"host": "0.0.0.0", "host": "0.0.0.0",
"healthCheckPath": "/health", "healthCheckPath": "/health",
"metricsPath": "/metrics", "metricsPath": "/metrics",
"shutdownTimeout": 30000, "shutdownTimeout": 30000,
"cors": { "cors": {
"enabled": true, "enabled": true,
"origin": "*", "origin": "*",
"credentials": true "credentials": true
} }
}, },
"database": { "database": {
"postgres": { "postgres": {
"enabled": true, "enabled": true,
"host": "localhost", "host": "localhost",
"port": 5432, "port": 5432,
"database": "trading_bot", "database": "trading_bot",
"user": "trading_user", "user": "trading_user",
"password": "trading_pass_dev", "password": "trading_pass_dev",
"ssl": false, "ssl": false,
"poolSize": 20, "poolSize": 20,
"connectionTimeout": 30000, "connectionTimeout": 30000,
"idleTimeout": 10000 "idleTimeout": 10000
}, },
"questdb": { "questdb": {
"host": "localhost", "host": "localhost",
"ilpPort": 9009, "ilpPort": 9009,
"httpPort": 9000, "httpPort": 9000,
"pgPort": 8812, "pgPort": 8812,
"database": "questdb", "database": "questdb",
"user": "admin", "user": "admin",
"password": "quest", "password": "quest",
"bufferSize": 65536, "bufferSize": 65536,
"flushInterval": 1000 "flushInterval": 1000
}, },
"mongodb": { "mongodb": {
"uri": "mongodb://trading_admin:trading_mongo_dev@localhost:27017/stock?authSource=admin", "uri": "mongodb://trading_admin:trading_mongo_dev@localhost:27017/stock?authSource=admin",
"database": "stock", "database": "stock",
"poolSize": 20 "poolSize": 20
}, },
"dragonfly": { "dragonfly": {
"host": "localhost", "host": "localhost",
"port": 6379, "port": 6379,
"db": 0, "db": 0,
"keyPrefix": "stock-bot:", "keyPrefix": "stock-bot:",
"maxRetries": 3, "maxRetries": 3,
"retryDelay": 100 "retryDelay": 100
} }
}, },
"log": { "log": {
"level": "info", "level": "info",
"format": "json", "format": "json",
"hideObject": false, "hideObject": false,
"loki": { "loki": {
"enabled": false, "enabled": false,
"host": "localhost", "host": "localhost",
"port": 3100, "port": 3100,
"labels": {} "labels": {}
} }
}, },
"redis": { "redis": {
"enabled": true, "enabled": true,
"host": "localhost", "host": "localhost",
"port": 6379, "port": 6379,
"db": 0 "db": 0
}, },
"queue": { "queue": {
"enabled": true, "enabled": true,
"redis": { "redis": {
"host": "localhost", "host": "localhost",
"port": 6379, "port": 6379,
"db": 1 "db": 1
}, },
"workers": 1, "workers": 1,
"concurrency": 1, "concurrency": 1,
"enableScheduledJobs": true, "enableScheduledJobs": true,
"delayWorkerStart": false, "delayWorkerStart": false,
"defaultJobOptions": { "defaultJobOptions": {
"attempts": 3, "attempts": 3,
"backoff": { "backoff": {
"type": "exponential", "type": "exponential",
"delay": 1000 "delay": 1000
}, },
"removeOnComplete": 100, "removeOnComplete": 100,
"removeOnFail": 50, "removeOnFail": 50,
"timeout": 300000 "timeout": 300000
} }
}, },
"http": { "http": {
"timeout": 30000, "timeout": 30000,
"retries": 3, "retries": 3,
"retryDelay": 1000, "retryDelay": 1000,
"userAgent": "StockBot/1.0", "userAgent": "StockBot/1.0",
"proxy": { "proxy": {
"enabled": false "enabled": false
} }
}, },
"webshare": { "webshare": {
"apiKey": "", "apiKey": "",
"apiUrl": "https://proxy.webshare.io/api/v2/", "apiUrl": "https://proxy.webshare.io/api/v2/",
"enabled": true "enabled": true
}, },
"browser": { "browser": {
"headless": true, "headless": true,
"timeout": 30000 "timeout": 30000
}, },
"proxy": { "proxy": {
"enabled": true, "enabled": true,
"cachePrefix": "proxy:", "cachePrefix": "proxy:",
"ttl": 3600, "ttl": 3600,
"webshare": { "webshare": {
"apiKey": "y8ay534rcbybdkk3evnzmt640xxfhy7252ce2t98", "apiKey": "y8ay534rcbybdkk3evnzmt640xxfhy7252ce2t98",
"apiUrl": "https://proxy.webshare.io/api/v2/" "apiUrl": "https://proxy.webshare.io/api/v2/"
} }
}, },
"providers": { "providers": {
"yahoo": { "yahoo": {
"name": "yahoo", "name": "yahoo",
"enabled": true, "enabled": true,
"priority": 1, "priority": 1,
"rateLimit": { "rateLimit": {
"maxRequests": 5, "maxRequests": 5,
"windowMs": 60000 "windowMs": 60000
}, },
"timeout": 30000, "timeout": 30000,
"baseUrl": "https://query1.finance.yahoo.com" "baseUrl": "https://query1.finance.yahoo.com"
}, },
"qm": { "qm": {
"name": "qm", "name": "qm",
"enabled": false, "enabled": false,
"priority": 2, "priority": 2,
"username": "", "username": "",
"password": "", "password": "",
"baseUrl": "https://app.quotemedia.com/quotetools", "baseUrl": "https://app.quotemedia.com/quotetools",
"webmasterId": "" "webmasterId": ""
}, },
"ib": { "ib": {
"name": "ib", "name": "ib",
"enabled": false, "enabled": false,
"priority": 3, "priority": 3,
"gateway": { "gateway": {
"host": "localhost", "host": "localhost",
"port": 5000, "port": 5000,
"clientId": 1 "clientId": 1
}, },
"marketDataType": "delayed" "marketDataType": "delayed"
}, },
"eod": { "eod": {
"name": "eod", "name": "eod",
"enabled": false, "enabled": false,
"priority": 4, "priority": 4,
"apiKey": "", "apiKey": "",
"baseUrl": "https://eodhistoricaldata.com/api", "baseUrl": "https://eodhistoricaldata.com/api",
"tier": "free" "tier": "free"
} }
}, },
"features": { "features": {
"realtime": true, "realtime": true,
"backtesting": true, "backtesting": true,
"paperTrading": true, "paperTrading": true,
"autoTrading": false, "autoTrading": false,
"historicalData": true, "historicalData": true,
"realtimeData": true, "realtimeData": true,
"fundamentalData": true, "fundamentalData": true,
"newsAnalysis": false, "newsAnalysis": false,
"notifications": false, "notifications": false,
"emailAlerts": false, "emailAlerts": false,
"smsAlerts": false, "smsAlerts": false,
"webhookAlerts": false, "webhookAlerts": false,
"technicalAnalysis": true, "technicalAnalysis": true,
"sentimentAnalysis": false, "sentimentAnalysis": false,
"patternRecognition": false, "patternRecognition": false,
"riskManagement": true, "riskManagement": true,
"positionSizing": true, "positionSizing": true,
"stopLoss": true, "stopLoss": true,
"takeProfit": true "takeProfit": true
}, },
"services": { "services": {
"dataIngestion": { "dataIngestion": {
"port": 2001, "port": 2001,
"workers": 4, "workers": 4,
"queues": { "queues": {
"ceo": { "concurrency": 2 }, "ceo": { "concurrency": 2 },
"webshare": { "concurrency": 1 }, "webshare": { "concurrency": 1 },
"qm": { "concurrency": 2 }, "qm": { "concurrency": 2 },
"ib": { "concurrency": 1 }, "ib": { "concurrency": 1 },
"proxy": { "concurrency": 1 } "proxy": { "concurrency": 1 }
}, },
"rateLimit": { "rateLimit": {
"enabled": true, "enabled": true,
"requestsPerSecond": 10 "requestsPerSecond": 10
} }
}, },
"dataPipeline": { "dataPipeline": {
"port": 2002, "port": 2002,
"workers": 2, "workers": 2,
"batchSize": 1000, "batchSize": 1000,
"processingInterval": 60000, "processingInterval": 60000,
"queues": { "queues": {
"exchanges": { "concurrency": 1 }, "exchanges": { "concurrency": 1 },
"symbols": { "concurrency": 2 } "symbols": { "concurrency": 2 }
}, },
"syncOptions": { "syncOptions": {
"maxRetries": 3, "maxRetries": 3,
"retryDelay": 5000, "retryDelay": 5000,
"timeout": 300000 "timeout": 300000
} }
}, },
"webApi": { "webApi": {
"port": 2003, "port": 2003,
"rateLimitPerMinute": 60, "rateLimitPerMinute": 60,
"cache": { "cache": {
"ttl": 300, "ttl": 300,
"checkPeriod": 60 "checkPeriod": 60
}, },
"cors": { "cors": {
"origins": ["http://localhost:3000", "http://localhost:4200"], "origins": ["http://localhost:3000", "http://localhost:4200"],
"credentials": true "credentials": true
} }
} }
} }
} }

22
apps/stock/config/config/development.json
View file

@ -1,11 +1,11 @@
{ {
"environment": "development", "environment": "development",
"log": { "log": {
"level": "debug", "level": "debug",
"format": "pretty" "format": "pretty"
}, },
"features": { "features": {
"autoTrading": false, "autoTrading": false,
"paperTrading": true "paperTrading": true
} }
} }

84
apps/stock/config/config/production.json
View file

@ -1,42 +1,42 @@
{ {
"environment": "production", "environment": "production",
"log": { "log": {
"level": "warn", "level": "warn",
"format": "json", "format": "json",
"loki": { "loki": {
"enabled": true, "enabled": true,
"host": "loki.production.example.com", "host": "loki.production.example.com",
"port": 3100 "port": 3100
} }
}, },
"database": { "database": {
"postgres": { "postgres": {
"host": "postgres.production.example.com", "host": "postgres.production.example.com",
"ssl": true, "ssl": true,
"poolSize": 50 "poolSize": 50
}, },
"questdb": { "questdb": {
"host": "questdb.production.example.com" "host": "questdb.production.example.com"
}, },
"mongodb": { "mongodb": {
"uri": "mongodb+srv://prod_user:prod_pass@cluster.mongodb.net/stock?retryWrites=true&w=majority", "uri": "mongodb+srv://prod_user:prod_pass@cluster.mongodb.net/stock?retryWrites=true&w=majority",
"poolSize": 50 "poolSize": 50
}, },
"dragonfly": { "dragonfly": {
"host": "redis.production.example.com", "host": "redis.production.example.com",
"password": "production_redis_password" "password": "production_redis_password"
} }
}, },
"queue": { "queue": {
"redis": { "redis": {
"host": "redis.production.example.com", "host": "redis.production.example.com",
"password": "production_redis_password" "password": "production_redis_password"
} }
}, },
"features": { "features": {
"autoTrading": true, "autoTrading": true,
"notifications": true, "notifications": true,
"emailAlerts": true, "emailAlerts": true,
"webhookAlerts": true "webhookAlerts": true
} }
} }

46
apps/stock/config/package.json
View file

@ -1,23 +1,23 @@
{ {
"name": "@stock-bot/stock-config", "name": "@stock-bot/stock-config",
"version": "1.0.0", "version": "1.0.0",
"description": "Stock trading bot configuration", "description": "Stock trading bot configuration",
"main": "dist/index.js", "main": "dist/index.js",
"types": "dist/index.d.ts", "types": "dist/index.d.ts",
"scripts": { "scripts": {
"build": "tsc", "build": "tsc",
"clean": "rm -rf dist", "clean": "rm -rf dist",
"dev": "tsc --watch", "dev": "tsc --watch",
"test": "jest", "test": "jest",
"lint": "eslint src --ext .ts" "lint": "eslint src --ext .ts"
}, },
"dependencies": { "dependencies": {
"@stock-bot/config": "*", "@stock-bot/config": "*",
"@stock-bot/logger": "*", "@stock-bot/logger": "*",
"zod": "^3.22.4" "zod": "^3.22.4"
}, },
"devDependencies": { "devDependencies": {
"@types/node": "^20.11.0", "@types/node": "^20.11.0",
"typescript": "^5.3.3" "typescript": "^5.3.3"
} }
} }

25
apps/stock/config/src/config-instance.ts
View file

@ -1,7 +1,7 @@
import { ConfigManager, createAppConfig } from '@stock-bot/config';
import { stockAppSchema, type StockAppConfig } from './schemas';
import * as path from 'path'; import * as path from 'path';
import { ConfigManager, createAppConfig } from '@stock-bot/config';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import { stockAppSchema, type StockAppConfig } from './schemas';
let configInstance: ConfigManager<StockAppConfig> | null = null; let configInstance: ConfigManager<StockAppConfig> | null = null;
@ -9,30 +9,35 @@ let configInstance: ConfigManager<StockAppConfig> | null = null;
* Initialize the stock application configuration * Initialize the stock application configuration
* @param serviceName - Optional service name to override port configuration * @param serviceName - Optional service name to override port configuration
*/ */
export function initializeStockConfig(serviceName?: 'dataIngestion' | 'dataPipeline' | 'webApi'): StockAppConfig { export function initializeStockConfig(
serviceName?: 'dataIngestion' | 'dataPipeline' | 'webApi'
): StockAppConfig {
try { try {
if (!configInstance) { if (!configInstance) {
configInstance = createAppConfig(stockAppSchema, { configInstance = createAppConfig(stockAppSchema, {
configPath: path.join(__dirname, '../config'), configPath: path.join(__dirname, '../config'),
}); });
} }
const config = configInstance.initialize(stockAppSchema); const config = configInstance.initialize(stockAppSchema);
// If a service name is provided, override the service port // If a service name is provided, override the service port
if (serviceName && config.services?.[serviceName]) { if (serviceName && config.services?.[serviceName]) {
const kebabName = serviceName.replace(/([A-Z])/g, '-$1').toLowerCase().replace(/^-/, ''); const kebabName = serviceName
.replace(/([A-Z])/g, '-$1')
.toLowerCase()
.replace(/^-/, '');
return { return {
...config, ...config,
service: { service: {
...config.service, ...config.service,
port: config.services[serviceName].port, port: config.services[serviceName].port,
name: serviceName, // Keep original for backward compatibility name: serviceName, // Keep original for backward compatibility
serviceName: kebabName // Standard kebab-case name serviceName: kebabName, // Standard kebab-case name
} },
}; };
} }
return config; return config;
} catch (error) { } catch (error) {
const logger = getLogger('stock-config'); const logger = getLogger('stock-config');
@ -85,4 +90,4 @@ export function isFeatureEnabled(feature: keyof StockAppConfig['features']): boo
*/ */
export function resetStockConfig(): void { export function resetStockConfig(): void {
configInstance = null; configInstance = null;
} }

30
apps/stock/config/src/index.ts
View file

@ -1,15 +1,15 @@
// Export schemas // Export schemas
export * from './schemas'; export * from './schemas';
// Export config instance functions // Export config instance functions
export { export {
initializeStockConfig, initializeStockConfig,
getStockConfig, getStockConfig,
getServiceConfig, getServiceConfig,
getProviderConfig, getProviderConfig,
isFeatureEnabled, isFeatureEnabled,
resetStockConfig, resetStockConfig,
} from './config-instance'; } from './config-instance';
// Re-export type for convenience // Re-export type for convenience
export type { StockAppConfig } from './schemas/stock-app.schema'; export type { StockAppConfig } from './schemas/stock-app.schema';

70
apps/stock/config/src/schemas/features.schema.ts
View file

@ -1,35 +1,35 @@
import { z } from 'zod'; import { z } from 'zod';
/** /**
* Feature flags for the stock trading application * Feature flags for the stock trading application
*/ */
export const featuresSchema = z.object({ export const featuresSchema = z.object({
// Trading features // Trading features
realtime: z.boolean().default(true), realtime: z.boolean().default(true),
backtesting: z.boolean().default(true), backtesting: z.boolean().default(true),
paperTrading: z.boolean().default(true), paperTrading: z.boolean().default(true),
autoTrading: z.boolean().default(false), autoTrading: z.boolean().default(false),
// Data features // Data features
historicalData: z.boolean().default(true), historicalData: z.boolean().default(true),
realtimeData: z.boolean().default(true), realtimeData: z.boolean().default(true),
fundamentalData: z.boolean().default(true), fundamentalData: z.boolean().default(true),
newsAnalysis: z.boolean().default(false), newsAnalysis: z.boolean().default(false),
// Notification features // Notification features
notifications: z.boolean().default(false), notifications: z.boolean().default(false),
emailAlerts: z.boolean().default(false), emailAlerts: z.boolean().default(false),
smsAlerts: z.boolean().default(false), smsAlerts: z.boolean().default(false),
webhookAlerts: z.boolean().default(false), webhookAlerts: z.boolean().default(false),
// Analysis features // Analysis features
technicalAnalysis: z.boolean().default(true), technicalAnalysis: z.boolean().default(true),
sentimentAnalysis: z.boolean().default(false), sentimentAnalysis: z.boolean().default(false),
patternRecognition: z.boolean().default(false), patternRecognition: z.boolean().default(false),
// Risk management // Risk management
riskManagement: z.boolean().default(true), riskManagement: z.boolean().default(true),
positionSizing: z.boolean().default(true), positionSizing: z.boolean().default(true),
stopLoss: z.boolean().default(true), stopLoss: z.boolean().default(true),
takeProfit: z.boolean().default(true), takeProfit: z.boolean().default(true),
}); });

6
apps/stock/config/src/schemas/index.ts
View file

@ -1,3 +1,3 @@
export * from './stock-app.schema'; export * from './stock-app.schema';
export * from './providers.schema'; export * from './providers.schema';
export * from './features.schema'; export * from './features.schema';

134
apps/stock/config/src/schemas/providers.schema.ts
View file

@ -1,67 +1,67 @@
import { z } from 'zod'; import { z } from 'zod';
// Base provider configuration // Base provider configuration
export const baseProviderConfigSchema = z.object({ export const baseProviderConfigSchema = z.object({
name: z.string(), name: z.string(),
enabled: z.boolean().default(true), enabled: z.boolean().default(true),
priority: z.number().default(0), priority: z.number().default(0),
rateLimit: z rateLimit: z
.object({ .object({
maxRequests: z.number().default(100), maxRequests: z.number().default(100),
windowMs: z.number().default(60000), windowMs: z.number().default(60000),
}) })
.optional(), .optional(),
timeout: z.number().default(30000), timeout: z.number().default(30000),
retries: z.number().default(3), retries: z.number().default(3),
}); });
// EOD Historical Data provider // EOD Historical Data provider
export const eodProviderConfigSchema = baseProviderConfigSchema.extend({ export const eodProviderConfigSchema = baseProviderConfigSchema.extend({
apiKey: z.string(), apiKey: z.string(),
baseUrl: z.string().default('https://eodhistoricaldata.com/api'), baseUrl: z.string().default('https://eodhistoricaldata.com/api'),
tier: z.enum(['free', 'fundamentals', 'all-in-one']).default('free'), tier: z.enum(['free', 'fundamentals', 'all-in-one']).default('free'),
}); });
// Interactive Brokers provider // Interactive Brokers provider
export const ibProviderConfigSchema = baseProviderConfigSchema.extend({ export const ibProviderConfigSchema = baseProviderConfigSchema.extend({
gateway: z.object({ gateway: z.object({
host: z.string().default('localhost'), host: z.string().default('localhost'),
port: z.number().default(5000), port: z.number().default(5000),
clientId: z.number().default(1), clientId: z.number().default(1),
}), }),
account: z.string().optional(), account: z.string().optional(),
marketDataType: z.enum(['live', 'delayed', 'frozen']).default('delayed'), marketDataType: z.enum(['live', 'delayed', 'frozen']).default('delayed'),
}); });
// QuoteMedia provider // QuoteMedia provider
export const qmProviderConfigSchema = baseProviderConfigSchema.extend({ export const qmProviderConfigSchema = baseProviderConfigSchema.extend({
username: z.string(), username: z.string(),
password: z.string(), password: z.string(),
baseUrl: z.string().default('https://app.quotemedia.com/quotetools'), baseUrl: z.string().default('https://app.quotemedia.com/quotetools'),
webmasterId: z.string(), webmasterId: z.string(),
}); });
// Yahoo Finance provider // Yahoo Finance provider
export const yahooProviderConfigSchema = baseProviderConfigSchema.extend({ export const yahooProviderConfigSchema = baseProviderConfigSchema.extend({
baseUrl: z.string().default('https://query1.finance.yahoo.com'), baseUrl: z.string().default('https://query1.finance.yahoo.com'),
cookieJar: z.boolean().default(true), cookieJar: z.boolean().default(true),
crumb: z.string().optional(), crumb: z.string().optional(),
}); });
// Combined provider configuration // Combined provider configuration
export const providersSchema = z.object({ export const providersSchema = z.object({
eod: eodProviderConfigSchema.optional(), eod: eodProviderConfigSchema.optional(),
ib: ibProviderConfigSchema.optional(), ib: ibProviderConfigSchema.optional(),
qm: qmProviderConfigSchema.optional(), qm: qmProviderConfigSchema.optional(),
yahoo: yahooProviderConfigSchema.optional(), yahoo: yahooProviderConfigSchema.optional(),
}); });
// Dynamic provider configuration type // Dynamic provider configuration type
export type ProviderName = 'eod' | 'ib' | 'qm' | 'yahoo'; export type ProviderName = 'eod' | 'ib' | 'qm' | 'yahoo';
export const providerSchemas = { export const providerSchemas = {
eod: eodProviderConfigSchema, eod: eodProviderConfigSchema,
ib: ibProviderConfigSchema, ib: ibProviderConfigSchema,
qm: qmProviderConfigSchema, qm: qmProviderConfigSchema,
yahoo: yahooProviderConfigSchema, yahoo: yahooProviderConfigSchema,
} as const; } as const;

168
apps/stock/config/src/schemas/stock-app.schema.ts
View file

@ -1,72 +1,96 @@
import { z } from 'zod'; import { z } from 'zod';
import { import {
baseAppSchema, baseAppSchema,
postgresConfigSchema, dragonflyConfigSchema,
mongodbConfigSchema, mongodbConfigSchema,
questdbConfigSchema, postgresConfigSchema,
dragonflyConfigSchema questdbConfigSchema,
} from '@stock-bot/config'; } from '@stock-bot/config';
import { providersSchema } from './providers.schema'; import { featuresSchema } from './features.schema';
import { featuresSchema } from './features.schema'; import { providersSchema } from './providers.schema';
/** /**
* Stock trading application configuration schema * Stock trading application configuration schema
*/ */
export const stockAppSchema = baseAppSchema.extend({ export const stockAppSchema = baseAppSchema.extend({
// Stock app uses all databases // Stock app uses all databases
database: z.object({ database: z.object({
postgres: postgresConfigSchema, postgres: postgresConfigSchema,
mongodb: mongodbConfigSchema, mongodb: mongodbConfigSchema,
questdb: questdbConfigSchema, questdb: questdbConfigSchema,
dragonfly: dragonflyConfigSchema, dragonfly: dragonflyConfigSchema,
}), }),
// Stock-specific providers // Stock-specific providers
providers: providersSchema, providers: providersSchema,
// Feature flags // Feature flags
features: featuresSchema, features: featuresSchema,
// Service-specific configurations // Service-specific configurations
services: z.object({ services: z
dataIngestion: z.object({ .object({
port: z.number().default(2001), dataIngestion: z
workers: z.number().default(4), .object({
queues: z.record(z.object({ port: z.number().default(2001),
concurrency: z.number().default(1), workers: z.number().default(4),
})).optional(), queues: z
rateLimit: z.object({ .record(
enabled: z.boolean().default(true), z.object({
requestsPerSecond: z.number().default(10), concurrency: z.number().default(1),
}).optional(), })
}).optional(), )
dataPipeline: z.object({ .optional(),
port: z.number().default(2002), rateLimit: z
workers: z.number().default(2), .object({
batchSize: z.number().default(1000), enabled: z.boolean().default(true),
processingInterval: z.number().default(60000), requestsPerSecond: z.number().default(10),
queues: z.record(z.object({ })
concurrency: z.number().default(1), .optional(),
})).optional(), })
syncOptions: z.object({ .optional(),
maxRetries: z.number().default(3), dataPipeline: z
retryDelay: z.number().default(5000), .object({
timeout: z.number().default(300000), port: z.number().default(2002),
}).optional(), workers: z.number().default(2),
}).optional(), batchSize: z.number().default(1000),
webApi: z.object({ processingInterval: z.number().default(60000),
port: z.number().default(2003), queues: z
rateLimitPerMinute: z.number().default(60), .record(
cache: z.object({ z.object({
ttl: z.number().default(300), concurrency: z.number().default(1),
checkPeriod: z.number().default(60), })
}).optional(), )
cors: z.object({ .optional(),
origins: z.array(z.string()).default(['http://localhost:3000']), syncOptions: z
credentials: z.boolean().default(true), .object({
}).optional(), maxRetries: z.number().default(3),
}).optional(), retryDelay: z.number().default(5000),
}).optional(), timeout: z.number().default(300000),
}); })
.optional(),
export type StockAppConfig = z.infer<typeof stockAppSchema>; })
.optional(),
webApi: z
.object({
port: z.number().default(2003),
rateLimitPerMinute: z.number().default(60),
cache: z
.object({
ttl: z.number().default(300),
checkPeriod: z.number().default(60),
})
.optional(),
cors: z
.object({
origins: z.array(z.string()).default(['http://localhost:3000']),
credentials: z.boolean().default(true),
})
.optional(),
})
.optional(),
})
.optional(),
});
export type StockAppConfig = z.infer<typeof stockAppSchema>;

28
apps/stock/config/tsconfig.json
View file

@ -1,15 +1,13 @@
{ {
"extends": "../../../tsconfig.json", "extends": "../../../tsconfig.json",
"compilerOptions": { "compilerOptions": {
"outDir": "./dist", "outDir": "./dist",
"rootDir": "./src", "rootDir": "./src",
"composite": true, "composite": true,
"declaration": true, "declaration": true,
"declarationMap": true "declarationMap": true
}, },
"include": ["src/**/*"], "include": ["src/**/*"],
"exclude": ["node_modules", "dist", "**/*.test.ts"], "exclude": ["node_modules", "dist", "**/*.test.ts"],
"references": [ "references": [{ "path": "../../../libs/core/config" }]
{ "path": "../../../libs/core/config" } }
]
}

12
apps/stock/data-ingestion/src/handlers/ceo/actions/process-individual-symbol.action.ts
View file

@ -95,10 +95,14 @@ export async function processIndividualSymbol(
await this.mongodb.batchUpsert('ceoShorts', shortData.positions, ['id']); await this.mongodb.batchUpsert('ceoShorts', shortData.positions, ['id']);
} }
await this.scheduleOperation('process-individual-symbol', { await this.scheduleOperation(
ceoId: ceoId, 'process-individual-symbol',
timestamp: latestSpielTime, {
}, {priority: 0}); ceoId: ceoId,
timestamp: latestSpielTime,
},
{ priority: 0 }
);
} }
this.logger.info( this.logger.info(

12
apps/stock/data-ingestion/src/handlers/ceo/actions/update-unique-symbols.action.ts
View file

@ -31,10 +31,14 @@ export async function updateUniqueSymbols(
let scheduledJobs = 0; let scheduledJobs = 0;
for (const symbol of uniqueSymbols) { for (const symbol of uniqueSymbols) {
// Schedule a job to process this individual symbol // Schedule a job to process this individual symbol
await this.scheduleOperation('process-individual-symbol', { await this.scheduleOperation(
ceoId: symbol.ceoId, 'process-individual-symbol',
symbol: symbol.symbol, {
}, {priority: 10 }); ceoId: symbol.ceoId,
symbol: symbol.symbol,
},
{ priority: 10 }
);
scheduledJobs++; scheduledJobs++;
// Add small delay to avoid overwhelming the queue // Add small delay to avoid overwhelming the queue

4
apps/stock/data-ingestion/src/handlers/ib/actions/fetch-exchanges-and-symbols.action.ts
View file

@ -1,6 +1,6 @@
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { fetchSession } from './fetch-session.action';
import { fetchExchanges } from './fetch-exchanges.action'; import { fetchExchanges } from './fetch-exchanges.action';
import { fetchSession } from './fetch-session.action';
import { fetchSymbols } from './fetch-symbols.action'; import { fetchSymbols } from './fetch-symbols.action';
export async function fetchExchangesAndSymbols(services: IServiceContainer): Promise<unknown> { export async function fetchExchangesAndSymbols(services: IServiceContainer): Promise<unknown> {
@ -38,5 +38,3 @@ export async function fetchExchangesAndSymbols(services: IServiceContainer): Pro
}; };
} }
} }

18
apps/stock/data-ingestion/src/handlers/ib/actions/fetch-exchanges.action.ts
View file

@ -1,4 +1,4 @@
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/types';
import { IB_CONFIG } from '../shared/config'; import { IB_CONFIG } from '../shared/config';
import { fetchSession } from './fetch-session.action'; import { fetchSession } from './fetch-session.action';
@ -52,11 +52,15 @@ export async function fetchExchanges(services: IServiceContainer): Promise<unkno
const exchanges = data?.exchanges || []; const exchanges = data?.exchanges || [];
services.logger.info('✅ Exchange data fetched successfully'); services.logger.info('✅ Exchange data fetched successfully');
services.logger.info('Saving IB exchanges to MongoDB...'); if (services.mongodb) {
await services.mongodb.batchUpsert('ibExchanges', exchanges, ['id', 'country_code']); services.logger.info('Saving IB exchanges to MongoDB...');
services.logger.info('✅ Exchange IB data saved to MongoDB:', { await services.mongodb.batchUpsert('ibExchanges', exchanges, ['id', 'country_code']);
count: exchanges.length, services.logger.info('✅ Exchange IB data saved to MongoDB:', {
}); count: exchanges.length,
});
} else {
services.logger.warn('MongoDB service not available, skipping data persistence');
}
return exchanges; return exchanges;
} catch (error) { } catch (error) {
@ -64,5 +68,3 @@ export async function fetchExchanges(services: IServiceContainer): Promise<unkno
return null; return null;
} }
} }

6
apps/stock/data-ingestion/src/handlers/ib/actions/fetch-session.action.ts
View file

@ -2,7 +2,9 @@ import { Browser } from '@stock-bot/browser';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { IB_CONFIG } from '../shared/config'; import { IB_CONFIG } from '../shared/config';
export async function fetchSession(services: IServiceContainer): Promise<Record<string, string> | undefined> { export async function fetchSession(
services: IServiceContainer
): Promise<Record<string, string> | undefined> {
try { try {
await Browser.initialize({ await Browser.initialize({
headless: true, headless: true,
@ -80,5 +82,3 @@ export async function fetchSession(services: IServiceContainer): Promise<Record<
return; return;
} }
} }

2
apps/stock/data-ingestion/src/handlers/ib/actions/fetch-symbols.action.ts
View file

@ -115,5 +115,3 @@ export async function fetchSymbols(services: IServiceContainer): Promise<unknown
return null; return null;
} }
} }

1
apps/stock/data-ingestion/src/handlers/ib/actions/index.ts
View file

@ -2,4 +2,3 @@ export { fetchSession } from './fetch-session.action';
export { fetchExchanges } from './fetch-exchanges.action'; export { fetchExchanges } from './fetch-exchanges.action';
export { fetchSymbols } from './fetch-symbols.action'; export { fetchSymbols } from './fetch-symbols.action';
export { fetchExchangesAndSymbols } from './fetch-exchanges-and-symbols.action'; export { fetchExchangesAndSymbols } from './fetch-exchanges-and-symbols.action';

4
apps/stock/data-ingestion/src/handlers/ib/ib.handler.ts
View file

@ -8,7 +8,7 @@ import {
import { fetchExchanges, fetchExchangesAndSymbols, fetchSession, fetchSymbols } from './actions'; import { fetchExchanges, fetchExchangesAndSymbols, fetchSession, fetchSymbols } from './actions';
@Handler('ib') @Handler('ib')
class IbHandler extends BaseHandler { export class IbHandler extends BaseHandler {
constructor(services: IServiceContainer) { constructor(services: IServiceContainer) {
super(services); super(services);
} }
@ -38,5 +38,3 @@ class IbHandler extends BaseHandler {
return fetchExchangesAndSymbols(this); return fetchExchangesAndSymbols(this);
} }
} }

1
apps/stock/data-ingestion/src/handlers/ib/shared/config.ts
View file

@ -21,4 +21,3 @@ export const IB_CONFIG = {
PRODUCT_COUNTRIES: ['CA', 'US'], PRODUCT_COUNTRIES: ['CA', 'US'],
PRODUCT_TYPES: ['STK'], PRODUCT_TYPES: ['STK'],
}; };

68
apps/stock/data-ingestion/src/handlers/index.ts
View file

@ -1,60 +1,48 @@
/** /**
* Handler auto-registration * Handler initialization for data-ingestion service
* Automatically discovers and registers all handlers * Uses explicit imports for bundling compatibility
*/ */
import type { IServiceContainer } from '@stock-bot/handlers';
import { autoRegisterHandlers } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
// Import handlers for bundling (ensures they're included in the build) import type { IServiceContainer } from '@stock-bot/types';
import './ceo/ceo.handler'; // Import handlers explicitly for bundling (ensures they're included in the build)
import './ib/ib.handler'; // These imports trigger the decorator metadata to be set
import './qm/qm.handler'; import { CeoHandler } from './ceo/ceo.handler';
import './webshare/webshare.handler'; import { IbHandler } from './ib/ib.handler';
import { QMHandler } from './qm/qm.handler';
import { WebShareHandler } from './webshare/webshare.handler';
// Add more handler imports as needed // Add more handler imports as needed
const logger = getLogger('handler-init'); const logger = getLogger('handler-init');
/** /**
* Initialize and register all handlers automatically * Initialize and register all handlers
* Note: The actual registration is now handled by the HandlerScanner in the DI container
* This function is kept for backward compatibility and explicit handler imports
*/ */
export async function initializeAllHandlers(serviceContainer: IServiceContainer): Promise<void> { export async function initializeAllHandlers(serviceContainer: IServiceContainer): Promise<void> {
try { try {
// Auto-register all handlers in this directory // The HandlerScanner in the DI container will handle the actual registration
const result = await autoRegisterHandlers(__dirname, serviceContainer, { // We just need to ensure handlers are imported so their decorators run
pattern: '.handler.',
exclude: ['test', 'spec'], const handlers = [CeoHandler, IbHandler, QMHandler, WebShareHandler];
dryRun: false,
serviceName: 'data-ingestion', logger.info('Handler imports loaded', {
count: handlers.length,
handlers: handlers.map(h => (h as any).__handlerName || h.name),
}); });
logger.info('Handler auto-registration complete', { // If the container has a handler scanner, we can manually register these
registered: result.registered, const scanner = (serviceContainer as any).handlerScanner;
failed: result.failed, if (scanner?.registerHandlerClass) {
}); for (const HandlerClass of handlers) {
scanner.registerHandlerClass(HandlerClass, { serviceName: 'data-ingestion' });
if (result.failed.length > 0) { }
logger.error('Some handlers failed to register', { failed: result.failed }); logger.info('Handlers registered with scanner');
} }
} catch (error) { } catch (error) {
logger.error('Handler auto-registration failed', { error }); logger.error('Handler initialization failed', { error });
// Fall back to manual registration
await manualHandlerRegistration(serviceContainer);
}
}
/**
* Manual fallback registration
*/
async function manualHandlerRegistration(_serviceContainer: IServiceContainer): Promise<void> {
logger.warn('Falling back to manual handler registration');
try {
logger.info('Manual handler registration complete');
} catch (error) {
logger.error('Manual handler registration failed', { error });
throw error; throw error;
} }
} }

10
apps/stock/data-ingestion/src/handlers/qm/actions/exchanges.action.ts
View file

@ -15,12 +15,18 @@ interface QMExchange {
export async function fetchExchanges(services: IServiceContainer): Promise<QMExchange[]> { export async function fetchExchanges(services: IServiceContainer): Promise<QMExchange[]> {
// Get exchanges from MongoDB // Get exchanges from MongoDB
const exchanges = await services.mongodb.collection<QMExchange>('qm_exchanges').find({}).toArray(); const exchanges = await services.mongodb
.collection<QMExchange>('qm_exchanges')
.find({})
.toArray();
return exchanges; return exchanges;
} }
export async function getExchangeByCode(services: IServiceContainer, code: string): Promise<QMExchange | null> { export async function getExchangeByCode(
services: IServiceContainer,
code: string
): Promise<QMExchange | null> {
// Get specific exchange by code // Get specific exchange by code
const exchange = await services.mongodb.collection<QMExchange>('qm_exchanges').findOne({ code }); const exchange = await services.mongodb.collection<QMExchange>('qm_exchanges').findOne({ code });

11
apps/stock/data-ingestion/src/handlers/qm/actions/symbols.action.ts
View file

@ -16,12 +16,19 @@ interface QMSymbol {
export async function searchSymbols(services: IServiceContainer): Promise<QMSymbol[]> { export async function searchSymbols(services: IServiceContainer): Promise<QMSymbol[]> {
// Get symbols from MongoDB // Get symbols from MongoDB
const symbols = await services.mongodb.collection<QMSymbol>('qm_symbols').find({}).limit(50).toArray(); const symbols = await services.mongodb
.collection<QMSymbol>('qm_symbols')
.find({})
.limit(50)
.toArray();
return symbols; return symbols;
} }
export async function fetchSymbolData(services: IServiceContainer, symbol: string): Promise<QMSymbol | null> { export async function fetchSymbolData(
services: IServiceContainer,
symbol: string
): Promise<QMSymbol | null> {
// Fetch data for a specific symbol // Fetch data for a specific symbol
const symbolData = await services.mongodb.collection<QMSymbol>('qm_symbols').findOne({ symbol }); const symbolData = await services.mongodb.collection<QMSymbol>('qm_symbols').findOne({ symbol });

2
apps/stock/data-ingestion/src/handlers/qm/qm.handler.ts
View file

@ -1,7 +1,7 @@
import { BaseHandler, Handler, type IServiceContainer } from '@stock-bot/handlers'; import { BaseHandler, Handler, type IServiceContainer } from '@stock-bot/handlers';
@Handler('qm') @Handler('qm')
class QMHandler extends BaseHandler { export class QMHandler extends BaseHandler {
constructor(services: IServiceContainer) { constructor(services: IServiceContainer) {
super(services); // Handler name read from @Handler decorator super(services); // Handler name read from @Handler decorator
} }

7
apps/stock/data-ingestion/src/handlers/webshare/webshare.handler.ts
View file

@ -4,17 +4,18 @@ import {
Operation, Operation,
QueueSchedule, QueueSchedule,
type ExecutionContext, type ExecutionContext,
type IServiceContainer type IServiceContainer,
} from '@stock-bot/handlers'; } from '@stock-bot/handlers';
@Handler('webshare') @Handler('webshare')
class WebShareHandler extends BaseHandler { export class WebShareHandler extends BaseHandler {
constructor(services: IServiceContainer) { constructor(services: IServiceContainer) {
super(services); super(services);
} }
@Operation('fetch-proxies') @Operation('fetch-proxies')
@QueueSchedule('0 */6 * * *', { // every 6 hours @QueueSchedule('0 */6 * * *', {
// every 6 hours
priority: 3, priority: 3,
immediately: false, // Don't run immediately since ProxyManager fetches on startup immediately: false, // Don't run immediately since ProxyManager fetches on startup
description: 'Refresh proxies from WebShare API', description: 'Refresh proxies from WebShare API',

18
apps/stock/data-ingestion/src/index.ts
View file

@ -3,15 +3,12 @@
* Simplified entry point using ServiceApplication framework * Simplified entry point using ServiceApplication framework
*/ */
import { initializeStockConfig, type StockAppConfig } from '@stock-bot/stock-config'; import { ServiceApplication } from '@stock-bot/di';
import {
ServiceApplication,
} from '@stock-bot/di';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import { initializeStockConfig, type StockAppConfig } from '@stock-bot/stock-config';
import { createRoutes } from './routes/create-routes';
// Local imports // Local imports
import { initializeAllHandlers } from './handlers'; import { initializeAllHandlers } from './handlers';
import { createRoutes } from './routes/create-routes';
// Initialize configuration with service-specific overrides // Initialize configuration with service-specific overrides
const config = initializeStockConfig('dataIngestion'); const config = initializeStockConfig('dataIngestion');
@ -44,7 +41,7 @@ const app = new ServiceApplication(
}, },
{ {
// Lifecycle hooks if needed // Lifecycle hooks if needed
onStarted: (_port) => { onStarted: _port => {
const logger = getLogger('data-ingestion'); const logger = getLogger('data-ingestion');
logger.info('Data ingestion service startup initiated with ServiceApplication framework'); logger.info('Data ingestion service startup initiated with ServiceApplication framework');
}, },
@ -54,7 +51,7 @@ const app = new ServiceApplication(
// Container factory function // Container factory function
async function createContainer(config: StockAppConfig) { async function createContainer(config: StockAppConfig) {
const { ServiceContainerBuilder } = await import('@stock-bot/di'); const { ServiceContainerBuilder } = await import('@stock-bot/di');
const container = await new ServiceContainerBuilder() const container = await new ServiceContainerBuilder()
.withConfig(config) .withConfig(config)
.withOptions({ .withOptions({
@ -67,14 +64,13 @@ async function createContainer(config: StockAppConfig) {
enableProxy: true, // Data ingestion needs proxy for rate limiting enableProxy: true, // Data ingestion needs proxy for rate limiting
}) })
.build(); // This automatically initializes services .build(); // This automatically initializes services
return container; return container;
} }
// Start the service // Start the service
app.start(createContainer, createRoutes, initializeAllHandlers).catch(error => { app.start(createContainer, createRoutes, initializeAllHandlers).catch(error => {
const logger = getLogger('data-ingestion'); const logger = getLogger('data-ingestion');
logger.fatal('Failed to start data service', { error }); logger.fatal('Failed to start data service', { error });
process.exit(1); process.exit(1);
}); });

37
apps/stock/data-ingestion/src/routes/market-data.routes.ts
View file

@ -2,9 +2,9 @@
* Market data routes * Market data routes
*/ */
import { Hono } from 'hono'; import { Hono } from 'hono';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import { processItems } from '@stock-bot/queue'; import { processItems } from '@stock-bot/queue';
import type { IServiceContainer } from '@stock-bot/handlers';
const logger = getLogger('market-data-routes'); const logger = getLogger('market-data-routes');
@ -22,7 +22,7 @@ export function createMarketDataRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ status: 'error', message: 'Queue manager not available' }, 503); return c.json({ status: 'error', message: 'Queue manager not available' }, 503);
} }
const queue = queueManager.getQueue('yahoo-finance'); const queue = queueManager.getQueue('yahoo-finance');
const job = await queue.add('live-data', { const job = await queue.add('live-data', {
handler: 'yahoo-finance', handler: 'yahoo-finance',
@ -57,7 +57,7 @@ export function createMarketDataRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ status: 'error', message: 'Queue manager not available' }, 503); return c.json({ status: 'error', message: 'Queue manager not available' }, 503);
} }
const queue = queueManager.getQueue('yahoo-finance'); const queue = queueManager.getQueue('yahoo-finance');
const job = await queue.add('historical-data', { const job = await queue.add('historical-data', {
handler: 'yahoo-finance', handler: 'yahoo-finance',
@ -110,18 +110,23 @@ export function createMarketDataRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ status: 'error', message: 'Queue manager not available' }, 503); return c.json({ status: 'error', message: 'Queue manager not available' }, 503);
} }
const result = await processItems(symbols, provider, { const result = await processItems(
handler: provider, symbols,
operation, provider,
totalDelayHours, {
useBatching, handler: provider,
batchSize, operation,
priority: 2, totalDelayHours,
retries: 2, useBatching,
removeOnComplete: 5, batchSize,
removeOnFail: 10, priority: 2,
}, queueManager); retries: 2,
removeOnComplete: 5,
removeOnFail: 10,
},
queueManager
);
return c.json({ return c.json({
status: 'success', status: 'success',
@ -139,4 +144,4 @@ export function createMarketDataRoutes(container: IServiceContainer) {
} }
// Legacy export for backward compatibility // Legacy export for backward compatibility
export const marketDataRoutes = createMarketDataRoutes({} as IServiceContainer); export const marketDataRoutes = createMarketDataRoutes({} as IServiceContainer);

6
apps/stock/data-ingestion/src/routes/queue.routes.ts
View file

@ -1,6 +1,6 @@
import { Hono } from 'hono'; import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('queue-routes'); const logger = getLogger('queue-routes');
@ -14,7 +14,7 @@ export function createQueueRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ status: 'error', message: 'Queue manager not available' }, 503); return c.json({ status: 'error', message: 'Queue manager not available' }, 503);
} }
const globalStats = await queueManager.getGlobalStats(); const globalStats = await queueManager.getGlobalStats();
return c.json({ return c.json({
@ -29,4 +29,4 @@ export function createQueueRoutes(container: IServiceContainer) {
}); });
return queue; return queue;
} }

68
apps/stock/data-pipeline/src/container-setup.ts
View file

@ -1,34 +1,34 @@
/** /**
* Service Container Setup for Data Pipeline * Service Container Setup for Data Pipeline
* Configures dependency injection for the data pipeline service * Configures dependency injection for the data pipeline service
*/ */
import type { IServiceContainer } from '@stock-bot/handlers'; import type { AppConfig } from '@stock-bot/config';
import { getLogger } from '@stock-bot/logger'; import type { IServiceContainer } from '@stock-bot/handlers';
import type { AppConfig } from '@stock-bot/config'; import { getLogger } from '@stock-bot/logger';
const logger = getLogger('data-pipeline-container'); const logger = getLogger('data-pipeline-container');
/** /**
* Configure the service container for data pipeline workloads * Configure the service container for data pipeline workloads
*/ */
export function setupServiceContainer( export function setupServiceContainer(
config: AppConfig, config: AppConfig,
container: IServiceContainer container: IServiceContainer
): IServiceContainer { ): IServiceContainer {
logger.info('Configuring data pipeline service container...'); logger.info('Configuring data pipeline service container...');
// Data pipeline specific configuration // Data pipeline specific configuration
// This service does more complex queries and transformations // This service does more complex queries and transformations
const poolSizes = { const poolSizes = {
mongodb: config.environment === 'production' ? 40 : 20, mongodb: config.environment === 'production' ? 40 : 20,
postgres: config.environment === 'production' ? 50 : 25, postgres: config.environment === 'production' ? 50 : 25,
cache: config.environment === 'production' ? 30 : 15, cache: config.environment === 'production' ? 30 : 15,
}; };
logger.info('Data pipeline pool sizes configured', poolSizes); logger.info('Data pipeline pool sizes configured', poolSizes);
// The container is already configured with connections // The container is already configured with connections
// Just return it with our logging // Just return it with our logging
return container; return container;
} }

224
apps/stock/data-pipeline/src/handlers/exchanges/exchanges.handler.ts
View file

@ -1,111 +1,113 @@
import { import {
BaseHandler, BaseHandler,
Handler, Handler,
Operation, Operation,
ScheduledOperation, ScheduledOperation,
type IServiceContainer, } from '@stock-bot/handlers';
} from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/types';
import { clearPostgreSQLData } from './operations/clear-postgresql-data.operations'; import { clearPostgreSQLData } from './operations/clear-postgresql-data.operations';
import { getSyncStatus } from './operations/enhanced-sync-status.operations'; import { getSyncStatus } from './operations/enhanced-sync-status.operations';
import { getExchangeStats } from './operations/exchange-stats.operations'; import { getExchangeStats } from './operations/exchange-stats.operations';
import { getProviderMappingStats } from './operations/provider-mapping-stats.operations'; import { getProviderMappingStats } from './operations/provider-mapping-stats.operations';
import { syncQMExchanges } from './operations/qm-exchanges.operations'; import { syncQMExchanges } from './operations/qm-exchanges.operations';
import { syncAllExchanges } from './operations/sync-all-exchanges.operations'; import { syncAllExchanges } from './operations/sync-all-exchanges.operations';
import { syncIBExchanges } from './operations/sync-ib-exchanges.operations'; import { syncIBExchanges } from './operations/sync-ib-exchanges.operations';
import { syncQMProviderMappings } from './operations/sync-qm-provider-mappings.operations'; import { syncQMProviderMappings } from './operations/sync-qm-provider-mappings.operations';
@Handler('exchanges') @Handler('exchanges')
class ExchangesHandler extends BaseHandler { class ExchangesHandler extends BaseHandler {
constructor(services: IServiceContainer) { constructor(services: IServiceContainer) {
super(services); super(services);
} }
/** /**
* Sync all exchanges - weekly full sync * Sync all exchanges - weekly full sync
*/ */
@Operation('sync-all-exchanges') @Operation('sync-all-exchanges')
@ScheduledOperation('sync-all-exchanges', '0 0 * * 0', { @ScheduledOperation('sync-all-exchanges', '0 0 * * 0', {
priority: 10, priority: 10,
description: 'Weekly full exchange sync on Sunday at midnight', description: 'Weekly full exchange sync on Sunday at midnight',
}) })
async syncAllExchanges(payload?: { clearFirst?: boolean }): Promise<unknown> { async syncAllExchanges(payload?: { clearFirst?: boolean }): Promise<unknown> {
const finalPayload = payload || { clearFirst: true }; const finalPayload = payload || { clearFirst: true };
this.log('info', 'Starting sync of all exchanges', finalPayload); this.log('info', 'Starting sync of all exchanges', finalPayload);
return syncAllExchanges(finalPayload, this.services); return syncAllExchanges(finalPayload, this.services);
} }
/** /**
* Sync exchanges from QuestionsAndMethods * Sync exchanges from QuestionsAndMethods
*/ */
@Operation('sync-qm-exchanges') @Operation('sync-qm-exchanges')
@ScheduledOperation('sync-qm-exchanges', '0 1 * * *', { @ScheduledOperation('sync-qm-exchanges', '0 1 * * *', {
priority: 5, priority: 5,
description: 'Daily sync of QM exchanges at 1 AM', description: 'Daily sync of QM exchanges at 1 AM',
}) })
async syncQMExchanges(): Promise<unknown> { async syncQMExchanges(): Promise<unknown> {
this.log('info', 'Starting QM exchanges sync...'); this.log('info', 'Starting QM exchanges sync...');
return syncQMExchanges({}, this.services); return syncQMExchanges({}, this.services);
} }
/** /**
* Sync exchanges from Interactive Brokers * Sync exchanges from Interactive Brokers
*/ */
@Operation('sync-ib-exchanges') @Operation('sync-ib-exchanges')
@ScheduledOperation('sync-ib-exchanges', '0 3 * * *', { @ScheduledOperation('sync-ib-exchanges', '0 3 * * *', {
priority: 3, priority: 3,
description: 'Daily sync of IB exchanges at 3 AM', description: 'Daily sync of IB exchanges at 3 AM',
}) })
async syncIBExchanges(): Promise<unknown> { async syncIBExchanges(): Promise<unknown> {
this.log('info', 'Starting IB exchanges sync...'); this.log('info', 'Starting IB exchanges sync...');
return syncIBExchanges({}, this.services); return syncIBExchanges({}, this.services);
} }
/** /**
* Sync provider mappings from QuestionsAndMethods * Sync provider mappings from QuestionsAndMethods
*/ */
@Operation('sync-qm-provider-mappings') @Operation('sync-qm-provider-mappings')
@ScheduledOperation('sync-qm-provider-mappings', '0 3 * * *', { @ScheduledOperation('sync-qm-provider-mappings', '0 3 * * *', {
priority: 7, priority: 7,
description: 'Daily sync of QM provider mappings at 3 AM', description: 'Daily sync of QM provider mappings at 3 AM',
}) })
async syncQMProviderMappings(): Promise<unknown> { async syncQMProviderMappings(): Promise<unknown> {
this.log('info', 'Starting QM provider mappings sync...'); this.log('info', 'Starting QM provider mappings sync...');
return syncQMProviderMappings({}, this.services); return syncQMProviderMappings({}, this.services);
} }
/** /**
* Clear PostgreSQL data - maintenance operation * Clear PostgreSQL data - maintenance operation
*/ */
@Operation('clear-postgresql-data') @Operation('clear-postgresql-data')
async clearPostgreSQLData(payload: { type?: 'exchanges' | 'provider_mappings' | 'all' }): Promise<unknown> { async clearPostgreSQLData(payload: {
this.log('warn', 'Clearing PostgreSQL data', payload); type?: 'exchanges' | 'provider_mappings' | 'all';
return clearPostgreSQLData(payload, this.services); }): Promise<unknown> {
} this.log('warn', 'Clearing PostgreSQL data', payload);
return clearPostgreSQLData(payload, this.services);
/** }
* Get exchange statistics
*/ /**
@Operation('get-exchange-stats') * Get exchange statistics
async getExchangeStats(): Promise<unknown> { */
this.log('info', 'Getting exchange statistics...'); @Operation('get-exchange-stats')
return getExchangeStats({}, this.services); async getExchangeStats(): Promise<unknown> {
} this.log('info', 'Getting exchange statistics...');
return getExchangeStats({}, this.services);
/** }
* Get provider mapping statistics
*/ /**
@Operation('get-provider-mapping-stats') * Get provider mapping statistics
async getProviderMappingStats(): Promise<unknown> { */
this.log('info', 'Getting provider mapping statistics...'); @Operation('get-provider-mapping-stats')
return getProviderMappingStats({}, this.services); async getProviderMappingStats(): Promise<unknown> {
} this.log('info', 'Getting provider mapping statistics...');
return getProviderMappingStats({}, this.services);
/** }
* Get enhanced sync status
*/ /**
@Operation('enhanced-sync-status') * Get enhanced sync status
async getEnhancedSyncStatus(): Promise<unknown> { */
this.log('info', 'Getting enhanced sync status...'); @Operation('enhanced-sync-status')
return getSyncStatus({}, this.services); async getEnhancedSyncStatus(): Promise<unknown> {
} this.log('info', 'Getting enhanced sync status...');
} return getSyncStatus({}, this.services);
}
}

2
apps/stock/data-pipeline/src/handlers/exchanges/operations/clear-postgresql-data.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads'; import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-clear-postgresql-data'); const logger = getLogger('enhanced-sync-clear-postgresql-data');

2
apps/stock/data-pipeline/src/handlers/exchanges/operations/enhanced-sync-status.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload, SyncStatus } from '../../../types/job-payloads'; import type { JobPayload, SyncStatus } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-status'); const logger = getLogger('enhanced-sync-status');

2
apps/stock/data-pipeline/src/handlers/exchanges/operations/exchange-stats.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads'; import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-exchange-stats'); const logger = getLogger('enhanced-sync-exchange-stats');

2
apps/stock/data-pipeline/src/handlers/exchanges/operations/provider-mapping-stats.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads'; import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-provider-mapping-stats'); const logger = getLogger('enhanced-sync-provider-mapping-stats');

12
apps/stock/data-pipeline/src/handlers/exchanges/operations/qm-exchanges.operations.ts
View file

@ -1,5 +1,5 @@
import type { IServiceContainer } from '@stock-bot/types';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import type { JobPayload } from '../../../types/job-payloads'; import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('sync-qm-exchanges'); const logger = getLogger('sync-qm-exchanges');
@ -62,7 +62,10 @@ interface Exchange {
visible: boolean; visible: boolean;
} }
async function findExchange(exchangeCode: string, postgresClient: IServiceContainer['postgres']): Promise<Exchange | null> { async function findExchange(
exchangeCode: string,
postgresClient: IServiceContainer['postgres']
): Promise<Exchange | null> {
const query = 'SELECT * FROM exchanges WHERE code = $1'; const query = 'SELECT * FROM exchanges WHERE code = $1';
const result = await postgresClient.query(query, [exchangeCode]); const result = await postgresClient.query(query, [exchangeCode]);
return result.rows[0] || null; return result.rows[0] || null;
@ -76,7 +79,10 @@ interface QMExchange {
countryCode?: string; countryCode?: string;
} }
async function createExchange(qmExchange: QMExchange, postgresClient: IServiceContainer['postgres']): Promise<void> { async function createExchange(
qmExchange: QMExchange,
postgresClient: IServiceContainer['postgres']
): Promise<void> {
const query = ` const query = `
INSERT INTO exchanges (code, name, country, currency, visible) INSERT INTO exchanges (code, name, country, currency, visible)
VALUES ($1, $2, $3, $4, $5) VALUES ($1, $2, $3, $4, $5)

14
apps/stock/data-pipeline/src/handlers/exchanges/operations/sync-all-exchanges.operations.ts
View file

@ -1,10 +1,13 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload, SyncResult } from '../../../types/job-payloads'; import type { JobPayload, SyncResult } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-all-exchanges'); const logger = getLogger('enhanced-sync-all-exchanges');
export async function syncAllExchanges(payload: JobPayload, container: IServiceContainer): Promise<SyncResult> { export async function syncAllExchanges(
payload: JobPayload,
container: IServiceContainer
): Promise<SyncResult> {
const clearFirst = payload.clearFirst || true; const clearFirst = payload.clearFirst || true;
logger.info('Starting comprehensive exchange sync...', { clearFirst }); logger.info('Starting comprehensive exchange sync...', { clearFirst });
@ -50,7 +53,6 @@ export async function syncAllExchanges(payload: JobPayload, container: IServiceC
} }
} }
async function clearPostgreSQLData(postgresClient: any): Promise<void> { async function clearPostgreSQLData(postgresClient: any): Promise<void> {
logger.info('Clearing existing PostgreSQL data...'); logger.info('Clearing existing PostgreSQL data...');
@ -141,7 +143,11 @@ async function createProviderExchangeMapping(
const postgresClient = container.postgres; const postgresClient = container.postgres;
// Check if mapping already exists // Check if mapping already exists
const existingMapping = await findProviderExchangeMapping(provider, providerExchangeCode, container); const existingMapping = await findProviderExchangeMapping(
provider,
providerExchangeCode,
container
);
if (existingMapping) { if (existingMapping) {
// Don't override existing mappings to preserve manual work // Don't override existing mappings to preserve manual work
return; return;

7
apps/stock/data-pipeline/src/handlers/exchanges/operations/sync-ib-exchanges.operations.ts
View file

@ -1,6 +1,6 @@
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import type { MasterExchange } from '@stock-bot/mongodb'; import type { MasterExchange } from '@stock-bot/mongodb';
import type { IServiceContainer } from '@stock-bot/handlers';
import type { JobPayload } from '../../../types/job-payloads'; import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('sync-ib-exchanges'); const logger = getLogger('sync-ib-exchanges');
@ -65,7 +65,10 @@ export async function syncIBExchanges(
/** /**
* Create or update master exchange record 1:1 from IB exchange * Create or update master exchange record 1:1 from IB exchange
*/ */
async function createOrUpdateMasterExchange(ibExchange: IBExchange, container: IServiceContainer): Promise<void> { async function createOrUpdateMasterExchange(
ibExchange: IBExchange,
container: IServiceContainer
): Promise<void> {
const mongoClient = container.mongodb; const mongoClient = container.mongodb;
const db = mongoClient.getDatabase(); const db = mongoClient.getDatabase();
const collection = db.collection<MasterExchange>('masterExchanges'); const collection = db.collection<MasterExchange>('masterExchanges');

9
apps/stock/data-pipeline/src/handlers/exchanges/operations/sync-qm-provider-mappings.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload, SyncResult } from '../../../types/job-payloads'; import type { JobPayload, SyncResult } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-qm-provider-mappings'); const logger = getLogger('enhanced-sync-qm-provider-mappings');
@ -86,7 +86,6 @@ export async function syncQMProviderMappings(
} }
} }
async function createProviderExchangeMapping( async function createProviderExchangeMapping(
provider: string, provider: string,
providerExchangeCode: string, providerExchangeCode: string,
@ -103,7 +102,11 @@ async function createProviderExchangeMapping(
const postgresClient = container.postgres; const postgresClient = container.postgres;
// Check if mapping already exists // Check if mapping already exists
const existingMapping = await findProviderExchangeMapping(provider, providerExchangeCode, container); const existingMapping = await findProviderExchangeMapping(
provider,
providerExchangeCode,
container
);
if (existingMapping) { if (existingMapping) {
// Don't override existing mappings to preserve manual work // Don't override existing mappings to preserve manual work
return; return;

83
apps/stock/data-pipeline/src/handlers/index.ts
View file

@ -1,42 +1,41 @@
/** /**
* Handler auto-registration for data pipeline service * Handler auto-registration for data pipeline service
* Automatically discovers and registers all handlers * Automatically discovers and registers all handlers
*/ */
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { autoRegisterHandlers } from '@stock-bot/handlers'; import { autoRegisterHandlers } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
// Import handlers for bundling (ensures they're included in the build)
// Import handlers for bundling (ensures they're included in the build) import './exchanges/exchanges.handler';
import './exchanges/exchanges.handler'; import './symbols/symbols.handler';
import './symbols/symbols.handler';
const logger = getLogger('pipeline-handler-init');
const logger = getLogger('pipeline-handler-init');
/**
/** * Initialize and register all handlers automatically
* Initialize and register all handlers automatically */
*/ export async function initializeAllHandlers(container: IServiceContainer): Promise<void> {
export async function initializeAllHandlers(container: IServiceContainer): Promise<void> { logger.info('Initializing data pipeline handlers...');
logger.info('Initializing data pipeline handlers...');
try {
try { // Auto-register all handlers in this directory
// Auto-register all handlers in this directory const result = await autoRegisterHandlers(__dirname, container, {
const result = await autoRegisterHandlers(__dirname, container, { pattern: '.handler.',
pattern: '.handler.', exclude: ['test', 'spec', '.old'],
exclude: ['test', 'spec', '.old'], dryRun: false,
dryRun: false, });
});
logger.info('Handler auto-registration complete', {
logger.info('Handler auto-registration complete', { registered: result.registered,
registered: result.registered, failed: result.failed,
failed: result.failed, });
});
if (result.failed.length > 0) {
if (result.failed.length > 0) { logger.error('Some handlers failed to register', { failed: result.failed });
logger.error('Some handlers failed to register', { failed: result.failed }); }
} } catch (error) {
} catch (error) { logger.error('Handler auto-registration failed', { error });
logger.error('Handler auto-registration failed', { error }); throw error;
throw error; }
} }
}

2
apps/stock/data-pipeline/src/handlers/symbols/operations/qm-symbols.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads'; import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('sync-qm-symbols'); const logger = getLogger('sync-qm-symbols');

2
apps/stock/data-pipeline/src/handlers/symbols/operations/sync-status.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads'; import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('sync-status'); const logger = getLogger('sync-status');

26
apps/stock/data-pipeline/src/handlers/symbols/operations/sync-symbols-from-provider.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload, SyncResult } from '../../../types/job-payloads'; import type { JobPayload, SyncResult } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-symbols-from-provider'); const logger = getLogger('enhanced-sync-symbols-from-provider');
@ -104,7 +104,11 @@ async function processSingleSymbol(
} }
// Find active provider exchange mapping // Find active provider exchange mapping
const providerMapping = await findActiveProviderExchangeMapping(provider, exchangeCode, container); const providerMapping = await findActiveProviderExchangeMapping(
provider,
exchangeCode,
container
);
if (!providerMapping) { if (!providerMapping) {
result.skipped++; result.skipped++;
@ -145,14 +149,22 @@ async function findActiveProviderExchangeMapping(
return result.rows[0] || null; return result.rows[0] || null;
} }
async function findSymbolByCodeAndExchange(symbol: string, exchangeId: string, container: IServiceContainer): Promise<any> { async function findSymbolByCodeAndExchange(
symbol: string,
exchangeId: string,
container: IServiceContainer
): Promise<any> {
const postgresClient = container.postgres; const postgresClient = container.postgres;
const query = 'SELECT * FROM symbols WHERE symbol = $1 AND exchange_id = $2'; const query = 'SELECT * FROM symbols WHERE symbol = $1 AND exchange_id = $2';
const result = await postgresClient.query(query, [symbol, exchangeId]); const result = await postgresClient.query(query, [symbol, exchangeId]);
return result.rows[0] || null; return result.rows[0] || null;
} }
async function createSymbol(symbol: any, exchangeId: string, container: IServiceContainer): Promise<string> { async function createSymbol(
symbol: any,
exchangeId: string,
container: IServiceContainer
): Promise<string> {
const postgresClient = container.postgres; const postgresClient = container.postgres;
const query = ` const query = `
INSERT INTO symbols (symbol, exchange_id, company_name, country, currency) INSERT INTO symbols (symbol, exchange_id, company_name, country, currency)
@ -171,7 +183,11 @@ async function createSymbol(symbol: any, exchangeId: string, container: IService
return result.rows[0].id; return result.rows[0].id;
} }
async function updateSymbol(symbolId: string, symbol: any, container: IServiceContainer): Promise<void> { async function updateSymbol(
symbolId: string,
symbol: any,
container: IServiceContainer
): Promise<void> {
const postgresClient = container.postgres; const postgresClient = container.postgres;
const query = ` const query = `
UPDATE symbols UPDATE symbols

139
apps/stock/data-pipeline/src/handlers/symbols/symbols.handler.ts
View file

@ -1,68 +1,71 @@
import { import {
BaseHandler, BaseHandler,
Handler, Handler,
Operation, Operation,
ScheduledOperation, ScheduledOperation,
type IServiceContainer, type IServiceContainer,
} from '@stock-bot/handlers'; } from '@stock-bot/handlers';
import { syncQMSymbols } from './operations/qm-symbols.operations'; import { syncQMSymbols } from './operations/qm-symbols.operations';
import { syncSymbolsFromProvider } from './operations/sync-symbols-from-provider.operations'; import { getSyncStatus } from './operations/sync-status.operations';
import { getSyncStatus } from './operations/sync-status.operations'; import { syncSymbolsFromProvider } from './operations/sync-symbols-from-provider.operations';
@Handler('symbols') @Handler('symbols')
class SymbolsHandler extends BaseHandler { class SymbolsHandler extends BaseHandler {
constructor(services: IServiceContainer) { constructor(services: IServiceContainer) {
super(services); super(services);
} }
/** /**
* Sync symbols from QuestionsAndMethods API * Sync symbols from QuestionsAndMethods API
*/ */
@ScheduledOperation('sync-qm-symbols', '0 2 * * *', { @ScheduledOperation('sync-qm-symbols', '0 2 * * *', {
priority: 5, priority: 5,
description: 'Daily sync of QM symbols at 2 AM', description: 'Daily sync of QM symbols at 2 AM',
}) })
async syncQMSymbols(): Promise<{ processed: number; created: number; updated: number }> { async syncQMSymbols(): Promise<{ processed: number; created: number; updated: number }> {
this.log('info', 'Starting QM symbols sync...'); this.log('info', 'Starting QM symbols sync...');
return syncQMSymbols({}, this.services); return syncQMSymbols({}, this.services);
} }
/** /**
* Sync symbols from specific provider * Sync symbols from specific provider
*/ */
@Operation('sync-symbols-qm') @Operation('sync-symbols-qm')
@ScheduledOperation('sync-symbols-qm', '0 4 * * *', { @ScheduledOperation('sync-symbols-qm', '0 4 * * *', {
priority: 5, priority: 5,
description: 'Daily sync of symbols from QM provider at 4 AM', description: 'Daily sync of symbols from QM provider at 4 AM',
}) })
async syncSymbolsQM(): Promise<unknown> { async syncSymbolsQM(): Promise<unknown> {
return this.syncSymbolsFromProvider({ provider: 'qm', clearFirst: false }); return this.syncSymbolsFromProvider({ provider: 'qm', clearFirst: false });
} }
@Operation('sync-symbols-eod') @Operation('sync-symbols-eod')
async syncSymbolsEOD(payload: { provider: string; clearFirst?: boolean }): Promise<unknown> { async syncSymbolsEOD(payload: { provider: string; clearFirst?: boolean }): Promise<unknown> {
return this.syncSymbolsFromProvider({ ...payload, provider: 'eod' }); return this.syncSymbolsFromProvider({ ...payload, provider: 'eod' });
} }
@Operation('sync-symbols-ib') @Operation('sync-symbols-ib')
async syncSymbolsIB(payload: { provider: string; clearFirst?: boolean }): Promise<unknown> { async syncSymbolsIB(payload: { provider: string; clearFirst?: boolean }): Promise<unknown> {
return this.syncSymbolsFromProvider({ ...payload, provider: 'ib' }); return this.syncSymbolsFromProvider({ ...payload, provider: 'ib' });
} }
/** /**
* Get sync status for symbols * Get sync status for symbols
*/ */
@Operation('sync-status') @Operation('sync-status')
async getSyncStatus(): Promise<unknown> { async getSyncStatus(): Promise<unknown> {
this.log('info', 'Getting symbol sync status...'); this.log('info', 'Getting symbol sync status...');
return getSyncStatus({}, this.services); return getSyncStatus({}, this.services);
} }
/** /**
* Internal method to sync symbols from a provider * Internal method to sync symbols from a provider
*/ */
private async syncSymbolsFromProvider(payload: { provider: string; clearFirst?: boolean }): Promise<unknown> { private async syncSymbolsFromProvider(payload: {
this.log('info', 'Syncing symbols from provider', { provider: payload.provider }); provider: string;
return syncSymbolsFromProvider(payload, this.services); clearFirst?: boolean;
} }): Promise<unknown> {
} this.log('info', 'Syncing symbols from provider', { provider: payload.provider });
return syncSymbolsFromProvider(payload, this.services);
}
}

17
apps/stock/data-pipeline/src/index.ts
View file

@ -3,14 +3,13 @@
* Simplified entry point using ServiceApplication framework * Simplified entry point using ServiceApplication framework
*/ */
import { initializeStockConfig } from '@stock-bot/stock-config';
import { ServiceApplication } from '@stock-bot/di'; import { ServiceApplication } from '@stock-bot/di';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import { initializeStockConfig } from '@stock-bot/stock-config';
// Local imports
import { initializeAllHandlers } from './handlers';
import { createRoutes } from './routes/create-routes'; import { createRoutes } from './routes/create-routes';
import { setupServiceContainer } from './container-setup'; import { setupServiceContainer } from './container-setup';
// Local imports
import { initializeAllHandlers } from './handlers';
// Initialize configuration with service-specific overrides // Initialize configuration with service-specific overrides
const config = initializeStockConfig('dataPipeline'); const config = initializeStockConfig('dataPipeline');
@ -43,12 +42,12 @@ const app = new ServiceApplication(
}, },
{ {
// Custom lifecycle hooks // Custom lifecycle hooks
onContainerReady: (container) => { onContainerReady: container => {
// Setup service-specific configuration // Setup service-specific configuration
const enhancedContainer = setupServiceContainer(config, container); const enhancedContainer = setupServiceContainer(config, container);
return enhancedContainer; return enhancedContainer;
}, },
onStarted: (_port) => { onStarted: _port => {
const logger = getLogger('data-pipeline'); const logger = getLogger('data-pipeline');
logger.info('Data pipeline service startup initiated with ServiceApplication framework'); logger.info('Data pipeline service startup initiated with ServiceApplication framework');
}, },
@ -59,7 +58,7 @@ const app = new ServiceApplication(
async function createContainer(config: any) { async function createContainer(config: any) {
const { ServiceContainerBuilder } = await import('@stock-bot/di'); const { ServiceContainerBuilder } = await import('@stock-bot/di');
const builder = new ServiceContainerBuilder(); const builder = new ServiceContainerBuilder();
const container = await builder const container = await builder
.withConfig(config) .withConfig(config)
.withOptions({ .withOptions({
@ -74,7 +73,7 @@ async function createContainer(config: any) {
skipInitialization: false, // Let builder handle initialization skipInitialization: false, // Let builder handle initialization
}) })
.build(); .build();
return container; return container;
} }
@ -83,4 +82,4 @@ app.start(createContainer, createRoutes, initializeAllHandlers).catch(error => {
const logger = getLogger('data-pipeline'); const logger = getLogger('data-pipeline');
logger.fatal('Failed to start data pipeline service', { error }); logger.fatal('Failed to start data pipeline service', { error });
process.exit(1); process.exit(1);
}); });

58
apps/stock/data-pipeline/src/routes/create-routes.ts
View file

@ -1,29 +1,29 @@
/** /**
* Route factory for data pipeline service * Route factory for data pipeline service
* Creates routes with access to the service container * Creates routes with access to the service container
*/ */
import { Hono } from 'hono'; import { Hono } from 'hono';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { healthRoutes } from './health.routes'; import { createEnhancedSyncRoutes } from './enhanced-sync.routes';
import { createSyncRoutes } from './sync.routes'; import { healthRoutes } from './health.routes';
import { createEnhancedSyncRoutes } from './enhanced-sync.routes'; import { createStatsRoutes } from './stats.routes';
import { createStatsRoutes } from './stats.routes'; import { createSyncRoutes } from './sync.routes';
export function createRoutes(container: IServiceContainer): Hono { export function createRoutes(container: IServiceContainer): Hono {
const app = new Hono(); const app = new Hono();
// Add container to context for all routes // Add container to context for all routes
app.use('*', async (c, next) => { app.use('*', async (c, next) => {
c.set('container', container); c.set('container', container);
await next(); await next();
}); });
// Mount routes // Mount routes
app.route('/health', healthRoutes); app.route('/health', healthRoutes);
app.route('/sync', createSyncRoutes(container)); app.route('/sync', createSyncRoutes(container));
app.route('/sync', createEnhancedSyncRoutes(container)); app.route('/sync', createEnhancedSyncRoutes(container));
app.route('/sync/stats', createStatsRoutes(container)); app.route('/sync/stats', createStatsRoutes(container));
return app; return app;
} }

14
apps/stock/data-pipeline/src/routes/enhanced-sync.routes.ts
View file

@ -1,6 +1,6 @@
import { Hono } from 'hono'; import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('enhanced-sync-routes'); const logger = getLogger('enhanced-sync-routes');
@ -15,7 +15,7 @@ export function createEnhancedSyncRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ success: false, error: 'Queue manager not available' }, 503); return c.json({ success: false, error: 'Queue manager not available' }, 503);
} }
const exchangesQueue = queueManager.getQueue('exchanges'); const exchangesQueue = queueManager.getQueue('exchanges');
const job = await exchangesQueue.addJob('sync-all-exchanges', { const job = await exchangesQueue.addJob('sync-all-exchanges', {
@ -40,7 +40,7 @@ export function createEnhancedSyncRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ success: false, error: 'Queue manager not available' }, 503); return c.json({ success: false, error: 'Queue manager not available' }, 503);
} }
const exchangesQueue = queueManager.getQueue('exchanges'); const exchangesQueue = queueManager.getQueue('exchanges');
const job = await exchangesQueue.addJob('sync-qm-provider-mappings', { const job = await exchangesQueue.addJob('sync-qm-provider-mappings', {
@ -69,7 +69,7 @@ export function createEnhancedSyncRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ success: false, error: 'Queue manager not available' }, 503); return c.json({ success: false, error: 'Queue manager not available' }, 503);
} }
const exchangesQueue = queueManager.getQueue('exchanges'); const exchangesQueue = queueManager.getQueue('exchanges');
const job = await exchangesQueue.addJob('sync-ib-exchanges', { const job = await exchangesQueue.addJob('sync-ib-exchanges', {
@ -98,7 +98,7 @@ export function createEnhancedSyncRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ success: false, error: 'Queue manager not available' }, 503); return c.json({ success: false, error: 'Queue manager not available' }, 503);
} }
const symbolsQueue = queueManager.getQueue('symbols'); const symbolsQueue = queueManager.getQueue('symbols');
const job = await symbolsQueue.addJob('sync-status', { const job = await symbolsQueue.addJob('sync-status', {
@ -124,7 +124,7 @@ export function createEnhancedSyncRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ success: false, error: 'Queue manager not available' }, 503); return c.json({ success: false, error: 'Queue manager not available' }, 503);
} }
const exchangesQueue = queueManager.getQueue('exchanges'); const exchangesQueue = queueManager.getQueue('exchanges');
const job = await exchangesQueue.addJob('clear-postgresql-data', { const job = await exchangesQueue.addJob('clear-postgresql-data', {
@ -148,4 +148,4 @@ export function createEnhancedSyncRoutes(container: IServiceContainer) {
}); });
return enhancedSync; return enhancedSync;
} }

8
apps/stock/data-pipeline/src/routes/stats.routes.ts
View file

@ -1,6 +1,6 @@
import { Hono } from 'hono'; import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('stats-routes'); const logger = getLogger('stats-routes');
@ -14,7 +14,7 @@ export function createStatsRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ error: 'Queue manager not available' }, 503); return c.json({ error: 'Queue manager not available' }, 503);
} }
const exchangesQueue = queueManager.getQueue('exchanges'); const exchangesQueue = queueManager.getQueue('exchanges');
const job = await exchangesQueue.addJob('get-exchange-stats', { const job = await exchangesQueue.addJob('get-exchange-stats', {
@ -38,7 +38,7 @@ export function createStatsRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ error: 'Queue manager not available' }, 503); return c.json({ error: 'Queue manager not available' }, 503);
} }
const exchangesQueue = queueManager.getQueue('exchanges'); const exchangesQueue = queueManager.getQueue('exchanges');
const job = await exchangesQueue.addJob('get-provider-mapping-stats', { const job = await exchangesQueue.addJob('get-provider-mapping-stats', {
@ -57,4 +57,4 @@ export function createStatsRoutes(container: IServiceContainer) {
}); });
return stats; return stats;
} }

10
apps/stock/data-pipeline/src/routes/sync.routes.ts
View file

@ -1,6 +1,6 @@
import { Hono } from 'hono'; import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('sync-routes'); const logger = getLogger('sync-routes');
@ -14,7 +14,7 @@ export function createSyncRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ success: false, error: 'Queue manager not available' }, 503); return c.json({ success: false, error: 'Queue manager not available' }, 503);
} }
const symbolsQueue = queueManager.getQueue('symbols'); const symbolsQueue = queueManager.getQueue('symbols');
const job = await symbolsQueue.addJob('sync-qm-symbols', { const job = await symbolsQueue.addJob('sync-qm-symbols', {
@ -39,7 +39,7 @@ export function createSyncRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ success: false, error: 'Queue manager not available' }, 503); return c.json({ success: false, error: 'Queue manager not available' }, 503);
} }
const exchangesQueue = queueManager.getQueue('exchanges'); const exchangesQueue = queueManager.getQueue('exchanges');
const job = await exchangesQueue.addJob('sync-qm-exchanges', { const job = await exchangesQueue.addJob('sync-qm-exchanges', {
@ -65,7 +65,7 @@ export function createSyncRoutes(container: IServiceContainer) {
if (!queueManager) { if (!queueManager) {
return c.json({ success: false, error: 'Queue manager not available' }, 503); return c.json({ success: false, error: 'Queue manager not available' }, 503);
} }
const symbolsQueue = queueManager.getQueue('symbols'); const symbolsQueue = queueManager.getQueue('symbols');
const job = await symbolsQueue.addJob('sync-symbols-from-provider', { const job = await symbolsQueue.addJob('sync-symbols-from-provider', {
@ -89,4 +89,4 @@ export function createSyncRoutes(container: IServiceContainer) {
}); });
return sync; return sync;
} }

172
apps/stock/package.json
View file

@ -1,91 +1,81 @@
{ {
"name": "@stock-bot/stock-app", "name": "@stock-bot/stock-app",
"version": "1.0.0", "version": "1.0.0",
"private": true, "private": true,
"description": "Stock trading bot application", "description": "Stock trading bot application",
"scripts": { "scripts": {
"dev": "turbo run dev", "dev": "turbo run dev",
"dev:ingestion": "cd data-ingestion && bun run dev", "dev:ingestion": "cd data-ingestion && bun run dev",
"dev:pipeline": "cd data-pipeline && bun run dev", "dev:pipeline": "cd data-pipeline && bun run dev",
"dev:api": "cd web-api && bun run dev", "dev:api": "cd web-api && bun run dev",
"dev:web": "cd web-app && bun run dev", "dev:web": "cd web-app && bun run dev",
"dev:backend": "turbo run dev --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-api\"", "dev:backend": "turbo run dev --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-api\"",
"dev:frontend": "turbo run dev --filter=\"@stock-bot/web-app\"", "dev:frontend": "turbo run dev --filter=\"@stock-bot/web-app\"",
"build": "echo 'Stock apps built via parent turbo'",
"build": "turbo run build", "build:config": "cd config && bun run build",
"build:config": "cd config && bun run build", "build:services": "turbo run build --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-*\"",
"build:services": "turbo run build --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-*\"", "build:ingestion": "cd data-ingestion && bun run build",
"build:ingestion": "cd data-ingestion && bun run build", "build:pipeline": "cd data-pipeline && bun run build",
"build:pipeline": "cd data-pipeline && bun run build", "build:api": "cd web-api && bun run build",
"build:api": "cd web-api && bun run build", "build:web": "cd web-app && bun run build",
"build:web": "cd web-app && bun run build", "start": "turbo run start --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-api\"",
"start:all": "turbo run start",
"start": "turbo run start --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-api\"", "start:ingestion": "cd data-ingestion && bun start",
"start:all": "turbo run start", "start:pipeline": "cd data-pipeline && bun start",
"start:ingestion": "cd data-ingestion && bun start", "start:api": "cd web-api && bun start",
"start:pipeline": "cd data-pipeline && bun start", "clean": "turbo run clean",
"start:api": "cd web-api && bun start", "clean:all": "turbo run clean && rm -rf node_modules",
"clean:ingestion": "cd data-ingestion && rm -rf dist node_modules",
"clean": "turbo run clean", "clean:pipeline": "cd data-pipeline && rm -rf dist node_modules",
"clean:all": "turbo run clean && rm -rf node_modules", "clean:api": "cd web-api && rm -rf dist node_modules",
"clean:ingestion": "cd data-ingestion && rm -rf dist node_modules", "clean:web": "cd web-app && rm -rf dist node_modules",
"clean:pipeline": "cd data-pipeline && rm -rf dist node_modules", "clean:config": "cd config && rm -rf dist node_modules",
"clean:api": "cd web-api && rm -rf dist node_modules", "test": "turbo run test",
"clean:web": "cd web-app && rm -rf dist node_modules", "test:all": "turbo run test",
"clean:config": "cd config && rm -rf dist node_modules", "test:config": "cd config && bun test",
"test:services": "turbo run test --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-*\"",
"test": "turbo run test", "test:ingestion": "cd data-ingestion && bun test",
"test:all": "turbo run test", "test:pipeline": "cd data-pipeline && bun test",
"test:config": "cd config && bun test", "test:api": "cd web-api && bun test",
"test:services": "turbo run test --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-*\"", "lint": "turbo run lint",
"test:ingestion": "cd data-ingestion && bun test", "lint:all": "turbo run lint",
"test:pipeline": "cd data-pipeline && bun test", "lint:config": "cd config && bun run lint",
"test:api": "cd web-api && bun test", "lint:services": "turbo run lint --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-*\"",
"lint:ingestion": "cd data-ingestion && bun run lint",
"lint": "turbo run lint", "lint:pipeline": "cd data-pipeline && bun run lint",
"lint:all": "turbo run lint", "lint:api": "cd web-api && bun run lint",
"lint:config": "cd config && bun run lint", "lint:web": "cd web-app && bun run lint",
"lint:services": "turbo run lint --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-*\"", "install:all": "bun install",
"lint:ingestion": "cd data-ingestion && bun run lint", "docker:build": "docker-compose build",
"lint:pipeline": "cd data-pipeline && bun run lint", "docker:up": "docker-compose up",
"lint:api": "cd web-api && bun run lint", "docker:down": "docker-compose down",
"lint:web": "cd web-app && bun run lint", "pm2:start": "pm2 start ecosystem.config.js",
"pm2:stop": "pm2 stop all",
"install:all": "bun install", "pm2:restart": "pm2 restart all",
"pm2:logs": "pm2 logs",
"docker:build": "docker-compose build", "pm2:status": "pm2 status",
"docker:up": "docker-compose up", "db:migrate": "cd data-ingestion && bun run db:migrate",
"docker:down": "docker-compose down", "db:seed": "cd data-ingestion && bun run db:seed",
"health:check": "bun scripts/health-check.js",
"pm2:start": "pm2 start ecosystem.config.js", "monitor": "bun run pm2:logs",
"pm2:stop": "pm2 stop all", "status": "bun run pm2:status"
"pm2:restart": "pm2 restart all", },
"pm2:logs": "pm2 logs", "devDependencies": {
"pm2:status": "pm2 status", "pm2": "^5.3.0",
"@types/node": "^20.11.0",
"db:migrate": "cd data-ingestion && bun run db:migrate", "typescript": "^5.3.3",
"db:seed": "cd data-ingestion && bun run db:seed", "turbo": "^2.5.4"
},
"health:check": "bun scripts/health-check.js", "workspaces": [
"monitor": "bun run pm2:logs", "config",
"status": "bun run pm2:status" "data-ingestion",
}, "data-pipeline",
"devDependencies": { "web-api",
"pm2": "^5.3.0", "web-app"
"@types/node": "^20.11.0", ],
"typescript": "^5.3.3", "engines": {
"turbo": "^2.5.4" "node": ">=18.0.0",
}, "bun": ">=1.1.0"
"workspaces": [ },
"config", "packageManager": "bun@1.1.12"
"data-ingestion", }
"data-pipeline",
"web-api",
"web-app"
],
"engines": {
"node": ">=18.0.0",
"bun": ">=1.1.0"
},
"packageManager": "bun@1.1.12"
}

36
apps/stock/tsconfig.json
View file

@ -1,18 +1,18 @@
{ {
"extends": "../../tsconfig.json", "extends": "../../tsconfig.json",
"compilerOptions": { "compilerOptions": {
"baseUrl": "../..", "baseUrl": "../..",
"paths": { "paths": {
"@stock-bot/*": ["libs/*/src"], "@stock-bot/*": ["libs/*/src"],
"@stock-bot/stock-config": ["apps/stock/config/src"], "@stock-bot/stock-config": ["apps/stock/config/src"],
"@stock-bot/stock-config/*": ["apps/stock/config/src/*"] "@stock-bot/stock-config/*": ["apps/stock/config/src/*"]
} }
}, },
"references": [ "references": [
{ "path": "./config" }, { "path": "./config" },
{ "path": "./data-ingestion" }, { "path": "./data-ingestion" },
{ "path": "./data-pipeline" }, { "path": "./data-pipeline" },
{ "path": "./web-api" }, { "path": "./web-api" },
{ "path": "./web-app" } { "path": "./web-app" }
] ]
} }

68
apps/stock/web-api/src/container-setup.ts
View file

@ -1,34 +1,34 @@
/** /**
* Service Container Setup for Web API * Service Container Setup for Web API
* Configures dependency injection for the web API service * Configures dependency injection for the web API service
*/ */
import type { IServiceContainer } from '@stock-bot/handlers'; import type { AppConfig } from '@stock-bot/config';
import { getLogger } from '@stock-bot/logger'; import type { IServiceContainer } from '@stock-bot/handlers';
import type { AppConfig } from '@stock-bot/config'; import { getLogger } from '@stock-bot/logger';
const logger = getLogger('web-api-container'); const logger = getLogger('web-api-container');
/** /**
* Configure the service container for web API workloads * Configure the service container for web API workloads
*/ */
export function setupServiceContainer( export function setupServiceContainer(
config: AppConfig, config: AppConfig,
container: IServiceContainer container: IServiceContainer
): IServiceContainer { ): IServiceContainer {
logger.info('Configuring web API service container...'); logger.info('Configuring web API service container...');
// Web API specific configuration // Web API specific configuration
// This service mainly reads data, so smaller pool sizes are fine // This service mainly reads data, so smaller pool sizes are fine
const poolSizes = { const poolSizes = {
mongodb: config.environment === 'production' ? 20 : 10, mongodb: config.environment === 'production' ? 20 : 10,
postgres: config.environment === 'production' ? 30 : 15, postgres: config.environment === 'production' ? 30 : 15,
cache: config.environment === 'production' ? 20 : 10, cache: config.environment === 'production' ? 20 : 10,
}; };
logger.info('Web API pool sizes configured', poolSizes); logger.info('Web API pool sizes configured', poolSizes);
// The container is already configured with connections // The container is already configured with connections
// Just return it with our logging // Just return it with our logging
return container; return container;
} }

11
apps/stock/web-api/src/index.ts
View file

@ -3,10 +3,9 @@
* Simplified entry point using ServiceApplication framework * Simplified entry point using ServiceApplication framework
*/ */
import { initializeStockConfig } from '@stock-bot/stock-config';
import { ServiceApplication } from '@stock-bot/di'; import { ServiceApplication } from '@stock-bot/di';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import { initializeStockConfig } from '@stock-bot/stock-config';
// Local imports // Local imports
import { createRoutes } from './routes/create-routes'; import { createRoutes } from './routes/create-routes';
@ -49,7 +48,7 @@ const app = new ServiceApplication(
}, },
{ {
// Custom lifecycle hooks // Custom lifecycle hooks
onStarted: (_port) => { onStarted: _port => {
const logger = getLogger('web-api'); const logger = getLogger('web-api');
logger.info('Web API service startup initiated with ServiceApplication framework'); logger.info('Web API service startup initiated with ServiceApplication framework');
}, },
@ -59,7 +58,7 @@ const app = new ServiceApplication(
// Container factory function // Container factory function
async function createContainer(config: any) { async function createContainer(config: any) {
const { ServiceContainerBuilder } = await import('@stock-bot/di'); const { ServiceContainerBuilder } = await import('@stock-bot/di');
const container = await new ServiceContainerBuilder() const container = await new ServiceContainerBuilder()
.withConfig(config) .withConfig(config)
.withOptions({ .withOptions({
@ -72,7 +71,7 @@ async function createContainer(config: any) {
enableProxy: false, // Web API doesn't need proxy enableProxy: false, // Web API doesn't need proxy
}) })
.build(); // This automatically initializes services .build(); // This automatically initializes services
return container; return container;
} }
@ -81,4 +80,4 @@ app.start(createContainer, createRoutes).catch(error => {
const logger = getLogger('web-api'); const logger = getLogger('web-api');
logger.fatal('Failed to start web API service', { error }); logger.fatal('Failed to start web API service', { error });
process.exit(1); process.exit(1);
}); });

4
apps/stock/web-api/src/routes/create-routes.ts
View file

@ -5,8 +5,8 @@
import { Hono } from 'hono'; import { Hono } from 'hono';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { createHealthRoutes } from './health.routes';
import { createExchangeRoutes } from './exchange.routes'; import { createExchangeRoutes } from './exchange.routes';
import { createHealthRoutes } from './health.routes';
import { createMonitoringRoutes } from './monitoring.routes'; import { createMonitoringRoutes } from './monitoring.routes';
import { createPipelineRoutes } from './pipeline.routes'; import { createPipelineRoutes } from './pipeline.routes';
@ -26,4 +26,4 @@ export function createRoutes(container: IServiceContainer): Hono {
app.route('/api/pipeline', pipelineRoutes); app.route('/api/pipeline', pipelineRoutes);
return app; return app;
} }

4
apps/stock/web-api/src/routes/exchange.routes.ts
View file

@ -2,8 +2,8 @@
* Exchange management routes - Refactored * Exchange management routes - Refactored
*/ */
import { Hono } from 'hono'; import { Hono } from 'hono';
import type { IServiceContainer } from '@stock-bot/types';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { createExchangeService } from '../services/exchange.service'; import { createExchangeService } from '../services/exchange.service';
import { createSuccessResponse, handleError } from '../utils/error-handler'; import { createSuccessResponse, handleError } from '../utils/error-handler';
import { import {
@ -259,4 +259,4 @@ export function createExchangeRoutes(container: IServiceContainer) {
}); });
return exchangeRoutes; return exchangeRoutes;
} }

9
apps/stock/web-api/src/routes/health.routes.ts
View file

@ -2,8 +2,8 @@
* Health check routes factory * Health check routes factory
*/ */
import { Hono } from 'hono'; import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('health-routes'); const logger = getLogger('health-routes');
@ -70,7 +70,10 @@ export function createHealthRoutes(container: IServiceContainer) {
health.checks.postgresql = { status: 'healthy', message: 'Connected and responsive' }; health.checks.postgresql = { status: 'healthy', message: 'Connected and responsive' };
logger.debug('PostgreSQL health check passed'); logger.debug('PostgreSQL health check passed');
} else { } else {
health.checks.postgresql = { status: 'unhealthy', message: 'PostgreSQL client not available' }; health.checks.postgresql = {
status: 'unhealthy',
message: 'PostgreSQL client not available',
};
logger.warn('PostgreSQL health check failed - client not available'); logger.warn('PostgreSQL health check failed - client not available');
} }
} catch (error) { } catch (error) {
@ -108,4 +111,4 @@ export function createHealthRoutes(container: IServiceContainer) {
} }
// Export legacy routes for backward compatibility during migration // Export legacy routes for backward compatibility during migration
export const healthRoutes = createHealthRoutes({} as IServiceContainer); export const healthRoutes = createHealthRoutes({} as IServiceContainer);

231
apps/stock/web-api/src/routes/monitoring.routes.ts
View file

@ -13,167 +13,200 @@ export function createMonitoringRoutes(container: IServiceContainer) {
/** /**
* Get overall system health * Get overall system health
*/ */
monitoring.get('/', async (c) => { monitoring.get('/', async c => {
try { try {
const health = await monitoringService.getSystemHealth(); const health = await monitoringService.getSystemHealth();
// Set appropriate status code based on health // Set appropriate status code based on health
const statusCode = health.status === 'healthy' ? 200 : const statusCode =
health.status === 'degraded' ? 503 : 500; health.status === 'healthy' ? 200 : health.status === 'degraded' ? 503 : 500;
return c.json(health, statusCode); return c.json(health, statusCode);
} catch (error) { } catch (error) {
return c.json({ return c.json(
status: 'error', {
message: 'Failed to retrieve system health', status: 'error',
error: error instanceof Error ? error.message : 'Unknown error', message: 'Failed to retrieve system health',
}, 500); error: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Get cache/Dragonfly statistics * Get cache/Dragonfly statistics
*/ */
monitoring.get('/cache', async (c) => { monitoring.get('/cache', async c => {
try { try {
const stats = await monitoringService.getCacheStats(); const stats = await monitoringService.getCacheStats();
return c.json(stats); return c.json(stats);
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve cache statistics', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve cache statistics',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Get queue statistics * Get queue statistics
*/ */
monitoring.get('/queues', async (c) => { monitoring.get('/queues', async c => {
try { try {
const stats = await monitoringService.getQueueStats(); const stats = await monitoringService.getQueueStats();
return c.json({ queues: stats }); return c.json({ queues: stats });
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve queue statistics', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve queue statistics',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Get specific queue statistics * Get specific queue statistics
*/ */
monitoring.get('/queues/:name', async (c) => { monitoring.get('/queues/:name', async c => {
try { try {
const queueName = c.req.param('name'); const queueName = c.req.param('name');
const stats = await monitoringService.getQueueStats(); const stats = await monitoringService.getQueueStats();
const queueStats = stats.find(q => q.name === queueName); const queueStats = stats.find(q => q.name === queueName);
if (!queueStats) { if (!queueStats) {
return c.json({ return c.json(
error: 'Queue not found', {
message: `Queue '${queueName}' does not exist`, error: 'Queue not found',
}, 404); message: `Queue '${queueName}' does not exist`,
},
404
);
} }
return c.json(queueStats); return c.json(queueStats);
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve queue statistics', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve queue statistics',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Get database statistics * Get database statistics
*/ */
monitoring.get('/databases', async (c) => { monitoring.get('/databases', async c => {
try { try {
const stats = await monitoringService.getDatabaseStats(); const stats = await monitoringService.getDatabaseStats();
return c.json({ databases: stats }); return c.json({ databases: stats });
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve database statistics', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve database statistics',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Get specific database statistics * Get specific database statistics
*/ */
monitoring.get('/databases/:type', async (c) => { monitoring.get('/databases/:type', async c => {
try { try {
const dbType = c.req.param('type') as 'postgres' | 'mongodb' | 'questdb'; const dbType = c.req.param('type') as 'postgres' | 'mongodb' | 'questdb';
const stats = await monitoringService.getDatabaseStats(); const stats = await monitoringService.getDatabaseStats();
const dbStats = stats.find(db => db.type === dbType); const dbStats = stats.find(db => db.type === dbType);
if (!dbStats) { if (!dbStats) {
return c.json({ return c.json(
error: 'Database not found', {
message: `Database type '${dbType}' not found or not enabled`, error: 'Database not found',
}, 404); message: `Database type '${dbType}' not found or not enabled`,
},
404
);
} }
return c.json(dbStats); return c.json(dbStats);
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve database statistics', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve database statistics',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Get service metrics * Get service metrics
*/ */
monitoring.get('/metrics', async (c) => { monitoring.get('/metrics', async c => {
try { try {
const metrics = await monitoringService.getServiceMetrics(); const metrics = await monitoringService.getServiceMetrics();
return c.json(metrics); return c.json(metrics);
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve service metrics', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve service metrics',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Get detailed cache info (Redis INFO command output) * Get detailed cache info (Redis INFO command output)
*/ */
monitoring.get('/cache/info', async (c) => { monitoring.get('/cache/info', async c => {
try { try {
if (!container.cache) { if (!container.cache) {
return c.json({ return c.json(
error: 'Cache not available', {
message: 'Cache service is not enabled', error: 'Cache not available',
}, 503); message: 'Cache service is not enabled',
},
503
);
} }
const info = await container.cache.info(); const info = await container.cache.info();
const stats = await monitoringService.getCacheStats(); const stats = await monitoringService.getCacheStats();
return c.json({ return c.json({
parsed: stats, parsed: stats,
raw: info, raw: info,
}); });
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve cache info', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve cache info',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Health check endpoint for monitoring * Health check endpoint for monitoring
*/ */
monitoring.get('/ping', (c) => { monitoring.get('/ping', c => {
return c.json({ return c.json({
status: 'ok', status: 'ok',
timestamp: new Date().toISOString(), timestamp: new Date().toISOString(),
service: 'monitoring', service: 'monitoring',
}); });
@ -182,78 +215,90 @@ export function createMonitoringRoutes(container: IServiceContainer) {
/** /**
* Get service status for all microservices * Get service status for all microservices
*/ */
monitoring.get('/services', async (c) => { monitoring.get('/services', async c => {
try { try {
const services = await monitoringService.getServiceStatus(); const services = await monitoringService.getServiceStatus();
return c.json({ services }); return c.json({ services });
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve service status', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve service status',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Get proxy statistics * Get proxy statistics
*/ */
monitoring.get('/proxies', async (c) => { monitoring.get('/proxies', async c => {
try { try {
const stats = await monitoringService.getProxyStats(); const stats = await monitoringService.getProxyStats();
return c.json(stats || { enabled: false }); return c.json(stats || { enabled: false });
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve proxy statistics', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve proxy statistics',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Get comprehensive system overview * Get comprehensive system overview
*/ */
monitoring.get('/overview', async (c) => { monitoring.get('/overview', async c => {
try { try {
const overview = await monitoringService.getSystemOverview(); const overview = await monitoringService.getSystemOverview();
return c.json(overview); return c.json(overview);
} catch (error) { } catch (error) {
return c.json({ return c.json(
error: 'Failed to retrieve system overview', {
message: error instanceof Error ? error.message : 'Unknown error', error: 'Failed to retrieve system overview',
}, 500); message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
} }
}); });
/** /**
* Test direct BullMQ queue access * Test direct BullMQ queue access
*/ */
monitoring.get('/test/queue/:name', async (c) => { monitoring.get('/test/queue/:name', async c => {
const queueName = c.req.param('name'); const queueName = c.req.param('name');
const { Queue } = await import('bullmq'); const { Queue } = await import('bullmq');
const connection = { const connection = {
host: 'localhost', host: 'localhost',
port: 6379, port: 6379,
db: 0, // All queues in DB 0 db: 0, // All queues in DB 0
}; };
const queue = new Queue(queueName, { connection }); const queue = new Queue(queueName, { connection });
try { try {
const counts = await queue.getJobCounts(); const counts = await queue.getJobCounts();
await queue.close(); await queue.close();
return c.json({ return c.json({
queueName, queueName,
counts counts,
}); });
} catch (error: any) { } catch (error: any) {
await queue.close(); await queue.close();
return c.json({ return c.json(
queueName, {
error: error.message queueName,
}, 500); error: error.message,
},
500
);
} }
}); });
return monitoring; return monitoring;
} }

2
apps/stock/web-api/src/routes/pipeline.routes.ts
View file

@ -132,4 +132,4 @@ export function createPipelineRoutes(container: IServiceContainer) {
}); });
return pipeline; return pipeline;
} }

4
apps/stock/web-api/src/services/exchange.service.ts
View file

@ -1,5 +1,5 @@
import type { IServiceContainer } from '@stock-bot/types';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { import {
CreateExchangeRequest, CreateExchangeRequest,
CreateProviderMappingRequest, CreateProviderMappingRequest,
@ -380,4 +380,4 @@ export class ExchangeService {
// Export function to create service instance with container // Export function to create service instance with container
export function createExchangeService(container: IServiceContainer): ExchangeService { export function createExchangeService(container: IServiceContainer): ExchangeService {
return new ExchangeService(container); return new ExchangeService(container);
} }

151
apps/stock/web-api/src/services/monitoring.service.ts
View file

@ -3,19 +3,19 @@
* Collects health and performance metrics from all system components * Collects health and performance metrics from all system components
*/ */
import * as os from 'os';
import type { IServiceContainer } from '@stock-bot/handlers'; import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import type { import type {
CacheStats, CacheStats,
QueueStats, DatabaseStats,
DatabaseStats, ProxyStats,
SystemHealth, QueueStats,
ServiceMetrics, ServiceMetrics,
ServiceStatus, ServiceStatus,
ProxyStats, SystemHealth,
SystemOverview SystemOverview,
} from '../types/monitoring.types'; } from '../types/monitoring.types';
import * as os from 'os';
export class MonitoringService { export class MonitoringService {
private readonly logger = getLogger('monitoring-service'); private readonly logger = getLogger('monitoring-service');
@ -46,7 +46,7 @@ export class MonitoringService {
// Get cache stats from the provider // Get cache stats from the provider
const cacheStats = this.container.cache.getStats(); const cacheStats = this.container.cache.getStats();
// Since we can't access Redis info directly, we'll use what's available // Since we can't access Redis info directly, we'll use what's available
return { return {
provider: 'dragonfly', provider: 'dragonfly',
@ -74,7 +74,7 @@ export class MonitoringService {
*/ */
async getQueueStats(): Promise<QueueStats[]> { async getQueueStats(): Promise<QueueStats[]> {
const stats: QueueStats[] = []; const stats: QueueStats[] = [];
try { try {
if (!this.container.queue) { if (!this.container.queue) {
this.logger.warn('No queue manager available'); this.logger.warn('No queue manager available');
@ -83,27 +83,27 @@ export class MonitoringService {
// Get all queue names from the SmartQueueManager // Get all queue names from the SmartQueueManager
const queueManager = this.container.queue as any; const queueManager = this.container.queue as any;
this.logger.debug('Queue manager type:', { this.logger.debug('Queue manager type:', {
type: queueManager.constructor.name, type: queueManager.constructor.name,
hasGetAllQueues: typeof queueManager.getAllQueues === 'function', hasGetAllQueues: typeof queueManager.getAllQueues === 'function',
hasQueues: !!queueManager.queues, hasQueues: !!queueManager.queues,
hasGetQueue: typeof queueManager.getQueue === 'function' hasGetQueue: typeof queueManager.getQueue === 'function',
}); });
// Always use the known queue names since web-api doesn't create worker queues // Always use the known queue names since web-api doesn't create worker queues
const handlerMapping = { const handlerMapping = {
'proxy': 'data-ingestion', proxy: 'data-ingestion',
'qm': 'data-ingestion', qm: 'data-ingestion',
'ib': 'data-ingestion', ib: 'data-ingestion',
'ceo': 'data-ingestion', ceo: 'data-ingestion',
'webshare': 'data-ingestion', webshare: 'data-ingestion',
'exchanges': 'data-pipeline', exchanges: 'data-pipeline',
'symbols': 'data-pipeline', symbols: 'data-pipeline',
}; };
const queueNames = Object.keys(handlerMapping); const queueNames = Object.keys(handlerMapping);
this.logger.debug('Using known queue names', { count: queueNames.length, names: queueNames }); this.logger.debug('Using known queue names', { count: queueNames.length, names: queueNames });
// Create BullMQ queues directly with the correct format // Create BullMQ queues directly with the correct format
for (const handlerName of queueNames) { for (const handlerName of queueNames) {
try { try {
@ -114,17 +114,17 @@ export class MonitoringService {
port: 6379, port: 6379,
db: 0, // All queues now in DB 0 db: 0, // All queues now in DB 0
}; };
// Get the service that owns this handler // Get the service that owns this handler
const serviceName = handlerMapping[handlerName as keyof typeof handlerMapping]; const serviceName = handlerMapping[handlerName as keyof typeof handlerMapping];
// Create BullMQ queue with the new naming format {service_handler} // Create BullMQ queue with the new naming format {service_handler}
const fullQueueName = `{${serviceName}_${handlerName}}`; const fullQueueName = `{${serviceName}_${handlerName}}`;
const bullQueue = new BullMQQueue(fullQueueName, { connection }); const bullQueue = new BullMQQueue(fullQueueName, { connection });
// Get stats directly from BullMQ // Get stats directly from BullMQ
const queueStats = await this.getQueueStatsForBullQueue(bullQueue, handlerName); const queueStats = await this.getQueueStatsForBullQueue(bullQueue, handlerName);
stats.push({ stats.push({
name: handlerName, name: handlerName,
connected: true, connected: true,
@ -134,7 +134,7 @@ export class MonitoringService {
concurrency: 1, concurrency: 1,
}, },
}); });
// Close the queue connection after getting stats // Close the queue connection after getting stats
await bullQueue.close(); await bullQueue.close();
} catch (error) { } catch (error) {
@ -167,7 +167,7 @@ export class MonitoringService {
try { try {
// BullMQ provides getJobCounts which returns all counts at once // BullMQ provides getJobCounts which returns all counts at once
const counts = await bullQueue.getJobCounts(); const counts = await bullQueue.getJobCounts();
return { return {
waiting: counts.waiting || 0, waiting: counts.waiting || 0,
active: counts.active || 0, active: counts.active || 0,
@ -184,11 +184,11 @@ export class MonitoringService {
try { try {
const [waiting, active, completed, failed, delayed, paused] = await Promise.all([ const [waiting, active, completed, failed, delayed, paused] = await Promise.all([
bullQueue.getWaitingCount(), bullQueue.getWaitingCount(),
bullQueue.getActiveCount(), bullQueue.getActiveCount(),
bullQueue.getCompletedCount(), bullQueue.getCompletedCount(),
bullQueue.getFailedCount(), bullQueue.getFailedCount(),
bullQueue.getDelayedCount(), bullQueue.getDelayedCount(),
bullQueue.getPausedCount ? bullQueue.getPausedCount() : 0 bullQueue.getPausedCount ? bullQueue.getPausedCount() : 0,
]); ]);
return { return {
@ -222,7 +222,7 @@ export class MonitoringService {
paused: stats.paused || 0, paused: stats.paused || 0,
}; };
} }
// Try individual count methods // Try individual count methods
const [waiting, active, completed, failed, delayed] = await Promise.all([ const [waiting, active, completed, failed, delayed] = await Promise.all([
this.safeGetCount(queue, 'getWaitingCount', 'getWaiting'), this.safeGetCount(queue, 'getWaitingCount', 'getWaiting'),
@ -252,7 +252,7 @@ export class MonitoringService {
if (queue[methodName] && typeof queue[methodName] === 'function') { if (queue[methodName] && typeof queue[methodName] === 'function') {
try { try {
const result = await queue[methodName](); const result = await queue[methodName]();
return Array.isArray(result) ? result.length : (result || 0); return Array.isArray(result) ? result.length : result || 0;
} catch (_e) { } catch (_e) {
// Continue to next method // Continue to next method
} }
@ -291,7 +291,7 @@ export class MonitoringService {
concurrency: queue.workers[0]?.concurrency || 1, concurrency: queue.workers[0]?.concurrency || 1,
}; };
} }
// Check queue manager for worker config // Check queue manager for worker config
if (queueManager.config?.defaultQueueOptions) { if (queueManager.config?.defaultQueueOptions) {
const options = queueManager.config.defaultQueueOptions; const options = queueManager.config.defaultQueueOptions;
@ -300,7 +300,7 @@ export class MonitoringService {
concurrency: options.concurrency || 1, concurrency: options.concurrency || 1,
}; };
} }
// Check for getWorkerCount method // Check for getWorkerCount method
if (queue.getWorkerCount && typeof queue.getWorkerCount === 'function') { if (queue.getWorkerCount && typeof queue.getWorkerCount === 'function') {
const count = queue.getWorkerCount(); const count = queue.getWorkerCount();
@ -312,7 +312,7 @@ export class MonitoringService {
} catch (_e) { } catch (_e) {
// Ignore // Ignore
} }
return undefined; return undefined;
} }
@ -331,12 +331,14 @@ export class MonitoringService {
// Get pool stats // Get pool stats
const pool = (this.container.postgres as any).pool; const pool = (this.container.postgres as any).pool;
const poolStats = pool ? { const poolStats = pool
size: pool.totalCount || 0, ? {
active: pool.idleCount || 0, size: pool.totalCount || 0,
idle: pool.waitingCount || 0, active: pool.idleCount || 0,
max: pool.options?.max || 0, idle: pool.waitingCount || 0,
} : undefined; max: pool.options?.max || 0,
}
: undefined;
stats.push({ stats.push({
type: 'postgres', type: 'postgres',
@ -365,7 +367,7 @@ export class MonitoringService {
const latency = Date.now() - startTime; const latency = Date.now() - startTime;
const serverStatus = await db.admin().serverStatus(); const serverStatus = await db.admin().serverStatus();
stats.push({ stats.push({
type: 'mongodb', type: 'mongodb',
name: 'MongoDB', name: 'MongoDB',
@ -393,9 +395,11 @@ export class MonitoringService {
try { try {
const startTime = Date.now(); const startTime = Date.now();
// QuestDB health check // QuestDB health check
const response = await fetch(`http://${process.env.QUESTDB_HOST || 'localhost'}:9000/exec?query=SELECT%201`); const response = await fetch(
`http://${process.env.QUESTDB_HOST || 'localhost'}:9000/exec?query=SELECT%201`
);
const latency = Date.now() - startTime; const latency = Date.now() - startTime;
stats.push({ stats.push({
type: 'questdb', type: 'questdb',
name: 'QuestDB', name: 'QuestDB',
@ -432,23 +436,22 @@ export class MonitoringService {
// Determine overall health status // Determine overall health status
const errors: string[] = []; const errors: string[] = [];
if (!cacheStats.connected) { if (!cacheStats.connected) {
errors.push('Cache service is disconnected'); errors.push('Cache service is disconnected');
} }
const disconnectedQueues = queueStats.filter(q => !q.connected); const disconnectedQueues = queueStats.filter(q => !q.connected);
if (disconnectedQueues.length > 0) { if (disconnectedQueues.length > 0) {
errors.push(`${disconnectedQueues.length} queue(s) are disconnected`); errors.push(`${disconnectedQueues.length} queue(s) are disconnected`);
} }
const disconnectedDbs = databaseStats.filter(db => !db.connected); const disconnectedDbs = databaseStats.filter(db => !db.connected);
if (disconnectedDbs.length > 0) { if (disconnectedDbs.length > 0) {
errors.push(`${disconnectedDbs.length} database(s) are disconnected`); errors.push(`${disconnectedDbs.length} database(s) are disconnected`);
} }
const status = errors.length === 0 ? 'healthy' : const status = errors.length === 0 ? 'healthy' : errors.length < 3 ? 'degraded' : 'unhealthy';
errors.length < 3 ? 'degraded' : 'unhealthy';
return { return {
status, status,
@ -478,7 +481,7 @@ export class MonitoringService {
*/ */
async getServiceMetrics(): Promise<ServiceMetrics> { async getServiceMetrics(): Promise<ServiceMetrics> {
const now = new Date().toISOString(); const now = new Date().toISOString();
return { return {
requestsPerSecond: { requestsPerSecond: {
timestamp: now, timestamp: now,
@ -517,12 +520,12 @@ export class MonitoringService {
private parseRedisInfo(info: string): Record<string, any> { private parseRedisInfo(info: string): Record<string, any> {
const result: Record<string, any> = {}; const result: Record<string, any> = {};
const sections = info.split('\r\n\r\n'); const sections = info.split('\r\n\r\n');
for (const section of sections) { for (const section of sections) {
const lines = section.split('\r\n'); const lines = section.split('\r\n');
const sectionName = lines[0]?.replace('# ', '') || 'general'; const sectionName = lines[0]?.replace('# ', '') || 'general';
result[sectionName] = {}; result[sectionName] = {};
for (let i = 1; i < lines.length; i++) { for (let i = 1; i < lines.length; i++) {
const [key, value] = lines[i].split(':'); const [key, value] = lines[i].split(':');
if (key && value) { if (key && value) {
@ -530,7 +533,7 @@ export class MonitoringService {
} }
} }
} }
return result; return result;
} }
@ -539,7 +542,7 @@ export class MonitoringService {
*/ */
async getServiceStatus(): Promise<ServiceStatus[]> { async getServiceStatus(): Promise<ServiceStatus[]> {
const services: ServiceStatus[] = []; const services: ServiceStatus[] = [];
// Define service endpoints // Define service endpoints
const serviceEndpoints = [ const serviceEndpoints = [
{ name: 'data-ingestion', port: 2001, path: '/health' }, { name: 'data-ingestion', port: 2001, path: '/health' },
@ -562,13 +565,13 @@ export class MonitoringService {
}); });
continue; continue;
} }
const startTime = Date.now(); const startTime = Date.now();
const response = await fetch(`http://localhost:${service.port}${service.path}`, { const response = await fetch(`http://localhost:${service.port}${service.path}`, {
signal: AbortSignal.timeout(5000), // 5 second timeout signal: AbortSignal.timeout(5000), // 5 second timeout
}); });
const _latency = Date.now() - startTime; const _latency = Date.now() - startTime;
if (response.ok) { if (response.ok) {
const data = await response.json(); const data = await response.json();
services.push({ services.push({
@ -629,28 +632,28 @@ export class MonitoringService {
// Get proxy data from cache using getRaw method // Get proxy data from cache using getRaw method
// The proxy manager uses cache:proxy: prefix, but web-api cache uses cache:api: // The proxy manager uses cache:proxy: prefix, but web-api cache uses cache:api:
const cacheProvider = this.container.cache; const cacheProvider = this.container.cache;
if (cacheProvider.getRaw) { if (cacheProvider.getRaw) {
// Use getRaw to access data with different cache prefix // Use getRaw to access data with different cache prefix
// The proxy manager now uses a global cache:proxy: prefix // The proxy manager now uses a global cache:proxy: prefix
this.logger.debug('Attempting to fetch proxy data from cache'); this.logger.debug('Attempting to fetch proxy data from cache');
const [cachedProxies, lastUpdateStr] = await Promise.all([ const [cachedProxies, lastUpdateStr] = await Promise.all([
cacheProvider.getRaw<any[]>('cache:proxy:active'), cacheProvider.getRaw<any[]>('cache:proxy:active'),
cacheProvider.getRaw<string>('cache:proxy:last-update') cacheProvider.getRaw<string>('cache:proxy:last-update'),
]); ]);
this.logger.debug('Proxy cache data retrieved', { this.logger.debug('Proxy cache data retrieved', {
hasProxies: !!cachedProxies, hasProxies: !!cachedProxies,
isArray: Array.isArray(cachedProxies), isArray: Array.isArray(cachedProxies),
proxyCount: cachedProxies ? cachedProxies.length : 0, proxyCount: cachedProxies ? cachedProxies.length : 0,
lastUpdate: lastUpdateStr lastUpdate: lastUpdateStr,
}); });
if (cachedProxies && Array.isArray(cachedProxies)) { if (cachedProxies && Array.isArray(cachedProxies)) {
const workingCount = cachedProxies.filter((p: any) => p.isWorking !== false).length; const workingCount = cachedProxies.filter((p: any) => p.isWorking !== false).length;
const failedCount = cachedProxies.filter((p: any) => p.isWorking === false).length; const failedCount = cachedProxies.filter((p: any) => p.isWorking === false).length;
return { return {
enabled: true, enabled: true,
totalProxies: cachedProxies.length, totalProxies: cachedProxies.length,
@ -662,7 +665,7 @@ export class MonitoringService {
} else { } else {
this.logger.debug('Cache provider does not support getRaw method'); this.logger.debug('Cache provider does not support getRaw method');
} }
// No cached data found - proxies might not be initialized yet // No cached data found - proxies might not be initialized yet
return { return {
enabled: true, enabled: true,
@ -672,7 +675,7 @@ export class MonitoringService {
}; };
} catch (cacheError) { } catch (cacheError) {
this.logger.debug('Could not retrieve proxy data from cache', { error: cacheError }); this.logger.debug('Could not retrieve proxy data from cache', { error: cacheError });
// Return basic stats if cache query fails // Return basic stats if cache query fails
return { return {
enabled: true, enabled: true,
@ -727,7 +730,7 @@ export class MonitoringService {
const idle = totalIdle / cpus.length; const idle = totalIdle / cpus.length;
const total = totalTick / cpus.length; const total = totalTick / cpus.length;
const usage = 100 - ~~(100 * idle / total); const usage = 100 - ~~((100 * idle) / total);
return { return {
usage, usage,
@ -742,21 +745,21 @@ export class MonitoringService {
private getSystemMemory() { private getSystemMemory() {
const totalMem = os.totalmem(); const totalMem = os.totalmem();
const freeMem = os.freemem(); const freeMem = os.freemem();
// On Linux, freeMem includes buffers/cache, but we want "available" memory // On Linux, freeMem includes buffers/cache, but we want "available" memory
// which better represents memory that can be used by applications // which better represents memory that can be used by applications
let availableMem = freeMem; let availableMem = freeMem;
// Try to read from /proc/meminfo for more accurate memory stats on Linux // Try to read from /proc/meminfo for more accurate memory stats on Linux
if (os.platform() === 'linux') { if (os.platform() === 'linux') {
try { try {
const fs = require('fs'); const fs = require('fs');
const meminfo = fs.readFileSync('/proc/meminfo', 'utf8'); const meminfo = fs.readFileSync('/proc/meminfo', 'utf8');
const lines = meminfo.split('\n'); const lines = meminfo.split('\n');
let memAvailable = 0; let memAvailable = 0;
let _memTotal = 0; let _memTotal = 0;
for (const line of lines) { for (const line of lines) {
if (line.startsWith('MemAvailable:')) { if (line.startsWith('MemAvailable:')) {
memAvailable = parseInt(line.split(/\s+/)[1], 10) * 1024; // Convert from KB to bytes memAvailable = parseInt(line.split(/\s+/)[1], 10) * 1024; // Convert from KB to bytes
@ -764,7 +767,7 @@ export class MonitoringService {
_memTotal = parseInt(line.split(/\s+/)[1], 10) * 1024; _memTotal = parseInt(line.split(/\s+/)[1], 10) * 1024;
} }
} }
if (memAvailable > 0) { if (memAvailable > 0) {
availableMem = memAvailable; availableMem = memAvailable;
} }
@ -773,7 +776,7 @@ export class MonitoringService {
this.logger.debug('Could not read /proc/meminfo', { error }); this.logger.debug('Could not read /proc/meminfo', { error });
} }
} }
const usedMem = totalMem - availableMem; const usedMem = totalMem - availableMem;
return { return {
@ -784,4 +787,4 @@ export class MonitoringService {
percentage: (usedMem / totalMem) * 100, percentage: (usedMem / totalMem) * 100,
}; };
} }
} }

2
apps/stock/web-api/src/services/pipeline.service.ts
View file

@ -332,4 +332,4 @@ export class PipelineService {
}; };
} }
} }
} }

2
apps/stock/web-api/src/types/monitoring.types.ts
View file

@ -124,4 +124,4 @@ export interface SystemOverview {
architecture: string; architecture: string;
hostname: string; hostname: string;
}; };
} }

1
apps/stock/web-app/src/components/ui/index.ts
View file

@ -3,4 +3,3 @@ export { Card, CardContent, CardHeader } from './Card';
export { DataTable } from './DataTable'; export { DataTable } from './DataTable';
export { Dialog, DialogContent, DialogHeader, DialogTitle } from './Dialog'; export { Dialog, DialogContent, DialogHeader, DialogTitle } from './Dialog';
export { StatCard } from './StatCard'; export { StatCard } from './StatCard';

1
apps/stock/web-app/src/features/exchanges/components/index.ts
View file

@ -1,4 +1,3 @@
export { AddProviderMappingDialog } from './AddProviderMappingDialog'; export { AddProviderMappingDialog } from './AddProviderMappingDialog';
export { AddExchangeDialog } from './AddExchangeDialog'; export { AddExchangeDialog } from './AddExchangeDialog';
export { DeleteExchangeDialog } from './DeleteExchangeDialog'; export { DeleteExchangeDialog } from './DeleteExchangeDialog';

2
apps/stock/web-app/src/features/exchanges/types/index.ts
View file

@ -133,4 +133,4 @@ interface BaseDialogProps {
export interface AddExchangeDialogProps extends BaseDialogProps { export interface AddExchangeDialogProps extends BaseDialogProps {
onCreateExchange: (request: CreateExchangeRequest) => Promise<void>; onCreateExchange: (request: CreateExchangeRequest) => Promise<void>;
} }

2
apps/stock/web-app/src/features/monitoring/components/index.ts
View file

@ -11,4 +11,4 @@ export { ProxyStatsCard } from './ProxyStatsCard';
export { StatusBadge, ConnectionStatus, HealthStatus, ServiceStatusIndicator } from './StatusBadge'; export { StatusBadge, ConnectionStatus, HealthStatus, ServiceStatusIndicator } from './StatusBadge';
export { MetricCard } from './MetricCard'; export { MetricCard } from './MetricCard';
export { ServiceCard } from './ServiceCard'; export { ServiceCard } from './ServiceCard';
export { DatabaseCard } from './DatabaseCard'; export { DatabaseCard } from './DatabaseCard';

2
apps/stock/web-app/src/features/monitoring/hooks/index.ts
View file

@ -2,4 +2,4 @@
* Monitoring hooks exports * Monitoring hooks exports
*/ */
export * from './useMonitoring'; export * from './useMonitoring';

30
apps/stock/web-app/src/features/monitoring/hooks/useMonitoring.ts
View file

@ -2,16 +2,16 @@
* Custom hook for monitoring data * Custom hook for monitoring data
*/ */
import { useState, useEffect, useCallback } from 'react'; import { useCallback, useEffect, useState } from 'react';
import { monitoringApi } from '../services/monitoringApi'; import { monitoringApi } from '../services/monitoringApi';
import type { import type {
SystemHealth, CacheStats,
CacheStats,
QueueStats,
DatabaseStats, DatabaseStats,
ServiceStatus,
ProxyStats, ProxyStats,
SystemOverview QueueStats,
ServiceStatus,
SystemHealth,
SystemOverview,
} from '../types'; } from '../types';
export function useSystemHealth(refreshInterval: number = 5000) { export function useSystemHealth(refreshInterval: number = 5000) {
@ -33,7 +33,7 @@ export function useSystemHealth(refreshInterval: number = 5000) {
useEffect(() => { useEffect(() => {
fetchData(); fetchData();
if (refreshInterval > 0) { if (refreshInterval > 0) {
const interval = setInterval(fetchData, refreshInterval); const interval = setInterval(fetchData, refreshInterval);
return () => clearInterval(interval); return () => clearInterval(interval);
@ -62,7 +62,7 @@ export function useCacheStats(refreshInterval: number = 5000) {
useEffect(() => { useEffect(() => {
fetchData(); fetchData();
if (refreshInterval > 0) { if (refreshInterval > 0) {
const interval = setInterval(fetchData, refreshInterval); const interval = setInterval(fetchData, refreshInterval);
return () => clearInterval(interval); return () => clearInterval(interval);
@ -91,7 +91,7 @@ export function useQueueStats(refreshInterval: number = 5000) {
useEffect(() => { useEffect(() => {
fetchData(); fetchData();
if (refreshInterval > 0) { if (refreshInterval > 0) {
const interval = setInterval(fetchData, refreshInterval); const interval = setInterval(fetchData, refreshInterval);
return () => clearInterval(interval); return () => clearInterval(interval);
@ -120,7 +120,7 @@ export function useDatabaseStats(refreshInterval: number = 5000) {
useEffect(() => { useEffect(() => {
fetchData(); fetchData();
if (refreshInterval > 0) { if (refreshInterval > 0) {
const interval = setInterval(fetchData, refreshInterval); const interval = setInterval(fetchData, refreshInterval);
return () => clearInterval(interval); return () => clearInterval(interval);
@ -149,7 +149,7 @@ export function useServiceStatus(refreshInterval: number = 5000) {
useEffect(() => { useEffect(() => {
fetchData(); fetchData();
if (refreshInterval > 0) { if (refreshInterval > 0) {
const interval = setInterval(fetchData, refreshInterval); const interval = setInterval(fetchData, refreshInterval);
return () => clearInterval(interval); return () => clearInterval(interval);
@ -178,7 +178,7 @@ export function useProxyStats(refreshInterval: number = 5000) {
useEffect(() => { useEffect(() => {
fetchData(); fetchData();
if (refreshInterval > 0) { if (refreshInterval > 0) {
const interval = setInterval(fetchData, refreshInterval); const interval = setInterval(fetchData, refreshInterval);
return () => clearInterval(interval); return () => clearInterval(interval);
@ -207,7 +207,7 @@ export function useSystemOverview(refreshInterval: number = 5000) {
useEffect(() => { useEffect(() => {
fetchData(); fetchData();
if (refreshInterval > 0) { if (refreshInterval > 0) {
const interval = setInterval(fetchData, refreshInterval); const interval = setInterval(fetchData, refreshInterval);
return () => clearInterval(interval); return () => clearInterval(interval);
@ -215,4 +215,4 @@ export function useSystemOverview(refreshInterval: number = 5000) {
}, [fetchData, refreshInterval]); }, [fetchData, refreshInterval]);
return { data, loading, error, refetch: fetchData }; return { data, loading, error, refetch: fetchData };
} }

2
apps/stock/web-app/src/features/monitoring/index.ts
View file

@ -5,4 +5,4 @@
export { MonitoringPage } from './MonitoringPage'; export { MonitoringPage } from './MonitoringPage';
export * from './types'; export * from './types';
export * from './hooks/useMonitoring'; export * from './hooks/useMonitoring';
export * from './services/monitoringApi'; export * from './services/monitoringApi';

14
apps/stock/web-app/src/features/monitoring/services/monitoringApi.ts
View file

@ -2,14 +2,14 @@
* Monitoring API Service * Monitoring API Service
*/ */
import type { import type {
SystemHealth, CacheStats,
CacheStats,
QueueStats,
DatabaseStats, DatabaseStats,
ServiceStatus,
ProxyStats, ProxyStats,
SystemOverview QueueStats,
ServiceStatus,
SystemHealth,
SystemOverview,
} from '../types'; } from '../types';
const API_BASE_URL = import.meta.env.VITE_API_BASE_URL || 'http://localhost:2003'; const API_BASE_URL = import.meta.env.VITE_API_BASE_URL || 'http://localhost:2003';
@ -125,4 +125,4 @@ export const monitoringApi = {
} }
return response.json(); return response.json();
}, },
}; };

2
apps/stock/web-app/src/features/monitoring/types/index.ts
View file

@ -117,4 +117,4 @@ export interface SystemOverview {
architecture: string; architecture: string;
hostname: string; hostname: string;
}; };
} }

90
apps/stock/web-app/src/features/monitoring/utils/formatters.ts
View file

@ -1,42 +1,48 @@
/** /**
* Common formatting utilities for monitoring components * Common formatting utilities for monitoring components
*/ */
export function formatUptime(ms: number): string { export function formatUptime(ms: number): string {
const seconds = Math.floor(ms / 1000); const seconds = Math.floor(ms / 1000);
const minutes = Math.floor(seconds / 60); const minutes = Math.floor(seconds / 60);
const hours = Math.floor(minutes / 60); const hours = Math.floor(minutes / 60);
const days = Math.floor(hours / 24); const days = Math.floor(hours / 24);
if (days > 0) {return `${days}d ${hours % 24}h`;} if (days > 0) {
if (hours > 0) {return `${hours}h ${minutes % 60}m`;} return `${days}d ${hours % 24}h`;
if (minutes > 0) {return `${minutes}m ${seconds % 60}s`;} }
return `${seconds}s`; if (hours > 0) {
} return `${hours}h ${minutes % 60}m`;
}
export function formatBytes(bytes: number): string { if (minutes > 0) {
const gb = bytes / 1024 / 1024 / 1024; return `${minutes}m ${seconds % 60}s`;
if (gb >= 1) { }
return gb.toFixed(2) + ' GB'; return `${seconds}s`;
} }
const mb = bytes / 1024 / 1024; export function formatBytes(bytes: number): string {
if (mb >= 1) { const gb = bytes / 1024 / 1024 / 1024;
return mb.toFixed(2) + ' MB'; if (gb >= 1) {
} return gb.toFixed(2) + ' GB';
}
const kb = bytes / 1024;
if (kb >= 1) { const mb = bytes / 1024 / 1024;
return kb.toFixed(2) + ' KB'; if (mb >= 1) {
} return mb.toFixed(2) + ' MB';
}
return bytes + ' B';
} const kb = bytes / 1024;
if (kb >= 1) {
export function formatNumber(num: number): string { return kb.toFixed(2) + ' KB';
return num.toLocaleString(); }
}
return bytes + ' B';
export function formatPercentage(value: number, decimals: number = 1): string { }
return `${value.toFixed(decimals)}%`;
} export function formatNumber(num: number): string {
return num.toLocaleString();
}
export function formatPercentage(value: number, decimals: number = 1): string {
return `${value.toFixed(decimals)}%`;
}

50
apps/stock/web-app/src/features/pipeline/hooks/usePipeline.ts
View file

@ -12,28 +12,29 @@ export function usePipeline() {
const [error, setError] = useState<string | null>(null); const [error, setError] = useState<string | null>(null);
const [lastJobResult, setLastJobResult] = useState<PipelineJobResult | null>(null); const [lastJobResult, setLastJobResult] = useState<PipelineJobResult | null>(null);
const executeOperation = useCallback(async ( const executeOperation = useCallback(
operation: () => Promise<PipelineJobResult> async (operation: () => Promise<PipelineJobResult>): Promise<boolean> => {
): Promise<boolean> => { try {
try { setLoading(true);
setLoading(true); setError(null);
setError(null); const result = await operation();
const result = await operation(); setLastJobResult(result);
setLastJobResult(result); if (!result.success) {
if (!result.success) { setError(result.error || 'Operation failed');
setError(result.error || 'Operation failed'); return false;
}
return true;
} catch (err) {
const errorMessage = err instanceof Error ? err.message : 'Unknown error occurred';
setError(errorMessage);
setLastJobResult({ success: false, error: errorMessage });
return false; return false;
} finally {
setLoading(false);
} }
return true; },
} catch (err) { []
const errorMessage = err instanceof Error ? err.message : 'Unknown error occurred'; );
setError(errorMessage);
setLastJobResult({ success: false, error: errorMessage });
return false;
} finally {
setLoading(false);
}
}, []);
// Symbol sync operations // Symbol sync operations
const syncQMSymbols = useCallback( const syncQMSymbols = useCallback(
@ -53,7 +54,7 @@ export function usePipeline() {
); );
const syncAllExchanges = useCallback( const syncAllExchanges = useCallback(
(clearFirst: boolean = false) => (clearFirst: boolean = false) =>
executeOperation(() => pipelineApi.syncAllExchanges(clearFirst)), executeOperation(() => pipelineApi.syncAllExchanges(clearFirst)),
[executeOperation] [executeOperation]
); );
@ -71,7 +72,7 @@ export function usePipeline() {
// Maintenance operations // Maintenance operations
const clearPostgreSQLData = useCallback( const clearPostgreSQLData = useCallback(
(dataType: DataClearType = 'all') => (dataType: DataClearType = 'all') =>
executeOperation(() => pipelineApi.clearPostgreSQLData(dataType)), executeOperation(() => pipelineApi.clearPostgreSQLData(dataType)),
[executeOperation] [executeOperation]
); );
@ -122,7 +123,8 @@ export function usePipeline() {
setError(result.error || 'Failed to get provider mapping stats'); setError(result.error || 'Failed to get provider mapping stats');
return null; return null;
} catch (err) { } catch (err) {
const errorMessage = err instanceof Error ? err.message : 'Failed to get provider mapping stats'; const errorMessage =
err instanceof Error ? err.message : 'Failed to get provider mapping stats';
setError(errorMessage); setError(errorMessage);
return null; return null;
} finally { } finally {
@ -156,4 +158,4 @@ export function usePipeline() {
getExchangeStats, getExchangeStats,
getProviderMappingStats, getProviderMappingStats,
}; };
} }

2
apps/stock/web-app/src/features/pipeline/index.ts
View file

@ -1,3 +1,3 @@
export { PipelinePage } from './PipelinePage'; export { PipelinePage } from './PipelinePage';
export * from './hooks/usePipeline'; export * from './hooks/usePipeline';
export * from './types'; export * from './types';

13
apps/stock/web-app/src/features/pipeline/services/pipelineApi.ts
View file

@ -1,16 +1,9 @@
import type { import type { DataClearType, PipelineJobResult, PipelineStatsResult } from '../types';
DataClearType,
PipelineJobResult,
PipelineStatsResult,
} from '../types';
const API_BASE_URL = import.meta.env.VITE_API_BASE_URL || 'http://localhost:2003'; const API_BASE_URL = import.meta.env.VITE_API_BASE_URL || 'http://localhost:2003';
class PipelineApiService { class PipelineApiService {
private async request<T = unknown>( private async request<T = unknown>(endpoint: string, options?: RequestInit): Promise<T> {
endpoint: string,
options?: RequestInit
): Promise<T> {
const url = `${API_BASE_URL}/pipeline${endpoint}`; const url = `${API_BASE_URL}/pipeline${endpoint}`;
const response = await fetch(url, { const response = await fetch(url, {
@ -79,4 +72,4 @@ class PipelineApiService {
} }
// Export singleton instance // Export singleton instance
export const pipelineApi = new PipelineApiService(); export const pipelineApi = new PipelineApiService();

3
apps/stock/web-app/src/features/pipeline/types/index.ts
View file

@ -32,7 +32,6 @@ export interface ProviderMappingStats {
coveragePercentage: number; coveragePercentage: number;
} }
export type DataClearType = 'exchanges' | 'provider_mappings' | 'all'; export type DataClearType = 'exchanges' | 'provider_mappings' | 'all';
export interface PipelineOperation { export interface PipelineOperation {
@ -44,4 +43,4 @@ export interface PipelineOperation {
category: 'sync' | 'stats' | 'maintenance'; category: 'sync' | 'stats' | 'maintenance';
dangerous?: boolean; dangerous?: boolean;
params?: Record<string, unknown>; params?: Record<string, unknown>;
} }

10
apps/stock/web-app/src/lib/constants.ts
View file

@ -1,13 +1,13 @@
import { import {
BuildingLibraryIcon, BuildingLibraryIcon,
ChartBarIcon, ChartBarIcon,
ChartPieIcon,
CircleStackIcon,
CogIcon, CogIcon,
DocumentTextIcon, DocumentTextIcon,
HomeIcon, HomeIcon,
PresentationChartLineIcon, PresentationChartLineIcon,
ServerStackIcon, ServerStackIcon,
CircleStackIcon,
ChartPieIcon,
} from '@heroicons/react/24/outline'; } from '@heroicons/react/24/outline';
export interface NavigationItem { export interface NavigationItem {
@ -23,13 +23,13 @@ export const navigation: NavigationItem[] = [
{ name: 'Portfolio', href: '/portfolio', icon: ChartBarIcon }, { name: 'Portfolio', href: '/portfolio', icon: ChartBarIcon },
{ name: 'Strategies', href: '/strategies', icon: DocumentTextIcon }, { name: 'Strategies', href: '/strategies', icon: DocumentTextIcon },
{ name: 'Analytics', href: '/analytics', icon: PresentationChartLineIcon }, { name: 'Analytics', href: '/analytics', icon: PresentationChartLineIcon },
{ {
name: 'System', name: 'System',
icon: ServerStackIcon, icon: ServerStackIcon,
children: [ children: [
{ name: 'Monitoring', href: '/system/monitoring', icon: ChartPieIcon }, { name: 'Monitoring', href: '/system/monitoring', icon: ChartPieIcon },
{ name: 'Pipeline', href: '/system/pipeline', icon: CircleStackIcon }, { name: 'Pipeline', href: '/system/pipeline', icon: CircleStackIcon },
] ],
}, },
{ name: 'Settings', href: '/settings', icon: CogIcon }, { name: 'Settings', href: '/settings', icon: CogIcon },
]; ];

48
bun.lock
View file

@ -189,6 +189,7 @@
"@stock-bot/browser": "workspace:*", "@stock-bot/browser": "workspace:*",
"@stock-bot/cache": "workspace:*", "@stock-bot/cache": "workspace:*",
"@stock-bot/config": "workspace:*", "@stock-bot/config": "workspace:*",
"@stock-bot/handler-registry": "workspace:*",
"@stock-bot/handlers": "workspace:*", "@stock-bot/handlers": "workspace:*",
"@stock-bot/logger": "workspace:*", "@stock-bot/logger": "workspace:*",
"@stock-bot/mongodb": "workspace:*", "@stock-bot/mongodb": "workspace:*",
@ -199,6 +200,7 @@
"@stock-bot/shutdown": "workspace:*", "@stock-bot/shutdown": "workspace:*",
"@stock-bot/types": "workspace:*", "@stock-bot/types": "workspace:*",
"awilix": "^12.0.5", "awilix": "^12.0.5",
"glob": "^10.0.0",
"hono": "^4.0.0", "hono": "^4.0.0",
"zod": "^3.23.8", "zod": "^3.23.8",
}, },
@ -220,12 +222,24 @@
"typescript": "^5.3.0", "typescript": "^5.3.0",
}, },
}, },
"libs/core/handler-registry": {
"name": "@stock-bot/handler-registry",
"version": "1.0.0",
"dependencies": {
"@stock-bot/types": "workspace:*",
},
"devDependencies": {
"@types/bun": "*",
"typescript": "*",
},
},
"libs/core/handlers": { "libs/core/handlers": {
"name": "@stock-bot/handlers", "name": "@stock-bot/handlers",
"version": "1.0.0", "version": "1.0.0",
"dependencies": { "dependencies": {
"@stock-bot/cache": "workspace:*", "@stock-bot/cache": "workspace:*",
"@stock-bot/config": "workspace:*", "@stock-bot/config": "workspace:*",
"@stock-bot/handler-registry": "workspace:*",
"@stock-bot/logger": "workspace:*", "@stock-bot/logger": "workspace:*",
"@stock-bot/types": "workspace:*", "@stock-bot/types": "workspace:*",
"@stock-bot/utils": "workspace:*", "@stock-bot/utils": "workspace:*",
@ -257,7 +271,7 @@
"version": "1.0.0", "version": "1.0.0",
"dependencies": { "dependencies": {
"@stock-bot/cache": "*", "@stock-bot/cache": "*",
"@stock-bot/handlers": "*", "@stock-bot/handler-registry": "*",
"@stock-bot/logger": "*", "@stock-bot/logger": "*",
"@stock-bot/types": "*", "@stock-bot/types": "*",
"bullmq": "^5.0.0", "bullmq": "^5.0.0",
@ -820,6 +834,8 @@
"@stock-bot/event-bus": ["@stock-bot/event-bus@workspace:libs/core/event-bus"], "@stock-bot/event-bus": ["@stock-bot/event-bus@workspace:libs/core/event-bus"],
"@stock-bot/handler-registry": ["@stock-bot/handler-registry@workspace:libs/core/handler-registry"],
"@stock-bot/handlers": ["@stock-bot/handlers@workspace:libs/core/handlers"], "@stock-bot/handlers": ["@stock-bot/handlers@workspace:libs/core/handlers"],
"@stock-bot/logger": ["@stock-bot/logger@workspace:libs/core/logger"], "@stock-bot/logger": ["@stock-bot/logger@workspace:libs/core/logger"],
@ -2144,7 +2160,7 @@
"side-channel-weakmap": ["side-channel-weakmap@1.0.2", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3", "side-channel-map": "^1.0.1" } }, "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A=="], "side-channel-weakmap": ["side-channel-weakmap@1.0.2", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3", "side-channel-map": "^1.0.1" } }, "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A=="],
"signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="], "signal-exit": ["signal-exit@4.1.0", "", {}, "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw=="],
"simple-concat": ["simple-concat@1.0.1", "", {}, "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q=="], "simple-concat": ["simple-concat@1.0.1", "", {}, "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q=="],
@ -2364,7 +2380,7 @@
"word-wrap": ["word-wrap@1.2.5", "", {}, "sha512-BN22B5eaMMI9UMtjrGd5g5eCYPpCPDUy0FJXbYsaT5zYxjFOckS53SQDE3pWkVoWpHXVb3BrYcEN4Twa55B5cA=="], "word-wrap": ["word-wrap@1.2.5", "", {}, "sha512-BN22B5eaMMI9UMtjrGd5g5eCYPpCPDUy0FJXbYsaT5zYxjFOckS53SQDE3pWkVoWpHXVb3BrYcEN4Twa55B5cA=="],
"wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], "wrap-ansi": ["wrap-ansi@8.1.0", "", { "dependencies": { "ansi-styles": "^6.1.0", "string-width": "^5.0.1", "strip-ansi": "^7.0.1" } }, "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ=="],
"wrap-ansi-cjs": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], "wrap-ansi-cjs": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="],
@ -2430,8 +2446,6 @@
"@isaacs/cliui/strip-ansi": ["strip-ansi@7.1.0", "", { "dependencies": { "ansi-regex": "^6.0.1" } }, "sha512-iq6eVVI64nQQTRYq2KtEg2d2uU7LElhTJwsH4YzIHZshxlgZms/wIc4VoDQTlG/IvVIrBKG06CrZnp0qv7hkcQ=="], "@isaacs/cliui/strip-ansi": ["strip-ansi@7.1.0", "", { "dependencies": { "ansi-regex": "^6.0.1" } }, "sha512-iq6eVVI64nQQTRYq2KtEg2d2uU7LElhTJwsH4YzIHZshxlgZms/wIc4VoDQTlG/IvVIrBKG06CrZnp0qv7hkcQ=="],
"@isaacs/cliui/wrap-ansi": ["wrap-ansi@8.1.0", "", { "dependencies": { "ansi-styles": "^6.1.0", "string-width": "^5.0.1", "strip-ansi": "^7.0.1" } }, "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ=="],
"@mongodb-js/oidc-plugin/express": ["express@4.21.2", "", { "dependencies": { "accepts": "~1.3.8", "array-flatten": "1.1.1", "body-parser": "1.20.3", "content-disposition": "0.5.4", "content-type": "~1.0.4", "cookie": "0.7.1", "cookie-signature": "1.0.6", "debug": "2.6.9", "depd": "2.0.0", "encodeurl": "~2.0.0", "escape-html": "~1.0.3", "etag": "~1.8.1", "finalhandler": "1.3.1", "fresh": "0.5.2", "http-errors": "2.0.0", "merge-descriptors": "1.0.3", "methods": "~1.1.2", "on-finished": "2.4.1", "parseurl": "~1.3.3", "path-to-regexp": "0.1.12", "proxy-addr": "~2.0.7", "qs": "6.13.0", "range-parser": "~1.2.1", "safe-buffer": "5.2.1", "send": "0.19.0", "serve-static": "1.16.2", "setprototypeof": "1.2.0", "statuses": "2.0.1", "type-is": "~1.6.18", "utils-merge": "1.0.1", "vary": "~1.1.2" } }, "sha512-28HqgMZAmih1Czt9ny7qr6ek2qddF4FclbMzwhCREB6OFfH+rXAnuNCwo1/wFvrtbgsQDb4kSbX9de9lFbrXnA=="], "@mongodb-js/oidc-plugin/express": ["express@4.21.2", "", { "dependencies": { "accepts": "~1.3.8", "array-flatten": "1.1.1", "body-parser": "1.20.3", "content-disposition": "0.5.4", "content-type": "~1.0.4", "cookie": "0.7.1", "cookie-signature": "1.0.6", "debug": "2.6.9", "depd": "2.0.0", "encodeurl": "~2.0.0", "escape-html": "~1.0.3", "etag": "~1.8.1", "finalhandler": "1.3.1", "fresh": "0.5.2", "http-errors": "2.0.0", "merge-descriptors": "1.0.3", "methods": "~1.1.2", "on-finished": "2.4.1", "parseurl": "~1.3.3", "path-to-regexp": "0.1.12", "proxy-addr": "~2.0.7", "qs": "6.13.0", "range-parser": "~1.2.1", "safe-buffer": "5.2.1", "send": "0.19.0", "serve-static": "1.16.2", "setprototypeof": "1.2.0", "statuses": "2.0.1", "type-is": "~1.6.18", "utils-merge": "1.0.1", "vary": "~1.1.2" } }, "sha512-28HqgMZAmih1Czt9ny7qr6ek2qddF4FclbMzwhCREB6OFfH+rXAnuNCwo1/wFvrtbgsQDb4kSbX9de9lFbrXnA=="],
"@pm2/agent/chalk": ["chalk@3.0.0", "", { "dependencies": { "ansi-styles": "^4.1.0", "supports-color": "^7.1.0" } }, "sha512-4D3B6Wf41KOYRFdszmDqMCGq5VV/uMAB273JILmO+3jAlh8X4qDtdtgCR3fxtbLEMzSx22QdhnDcJvu2u1fVwg=="], "@pm2/agent/chalk": ["chalk@3.0.0", "", { "dependencies": { "ansi-styles": "^4.1.0", "supports-color": "^7.1.0" } }, "sha512-4D3B6Wf41KOYRFdszmDqMCGq5VV/uMAB273JILmO+3jAlh8X4qDtdtgCR3fxtbLEMzSx22QdhnDcJvu2u1fVwg=="],
@ -2450,6 +2464,8 @@
"@pm2/io/semver": ["semver@7.5.4", "", { "dependencies": { "lru-cache": "^6.0.0" }, "bin": { "semver": "bin/semver.js" } }, "sha512-1bCSESV6Pv+i21Hvpxp3Dx+pSD8lIPt8uVjRrxAUt/nbswYc+tK6Y2btiULjd4+fnq15PX+nqQDC7Oft7WkwcA=="], "@pm2/io/semver": ["semver@7.5.4", "", { "dependencies": { "lru-cache": "^6.0.0" }, "bin": { "semver": "bin/semver.js" } }, "sha512-1bCSESV6Pv+i21Hvpxp3Dx+pSD8lIPt8uVjRrxAUt/nbswYc+tK6Y2btiULjd4+fnq15PX+nqQDC7Oft7WkwcA=="],
"@pm2/io/signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
"@pm2/io/tslib": ["tslib@1.9.3", "", {}, "sha512-4krF8scpejhaOgqzBEcGM7yDIEfi0/8+8zDRZhNZZ2kjmHJ4hv3zCbQWxoJGz1iw5U0Jl0nma13xzHXcncMavQ=="], "@pm2/io/tslib": ["tslib@1.9.3", "", {}, "sha512-4krF8scpejhaOgqzBEcGM7yDIEfi0/8+8zDRZhNZZ2kjmHJ4hv3zCbQWxoJGz1iw5U0Jl0nma13xzHXcncMavQ=="],
"@pm2/js-api/async": ["async@2.6.4", "", { "dependencies": { "lodash": "^4.17.14" } }, "sha512-mzo5dfJYwAn29PeiJ0zvwTo04zj8HDJj0Mn8TD7sno7q12prdbnasKJHhkm2c1LgrhlJ0teaea8860oxi51mGA=="], "@pm2/js-api/async": ["async@2.6.4", "", { "dependencies": { "lodash": "^4.17.14" } }, "sha512-mzo5dfJYwAn29PeiJ0zvwTo04zj8HDJj0Mn8TD7sno7q12prdbnasKJHhkm2c1LgrhlJ0teaea8860oxi51mGA=="],
@ -2516,6 +2532,8 @@
"cli-tableau/chalk": ["chalk@3.0.0", "", { "dependencies": { "ansi-styles": "^4.1.0", "supports-color": "^7.1.0" } }, "sha512-4D3B6Wf41KOYRFdszmDqMCGq5VV/uMAB273JILmO+3jAlh8X4qDtdtgCR3fxtbLEMzSx22QdhnDcJvu2u1fVwg=="], "cli-tableau/chalk": ["chalk@3.0.0", "", { "dependencies": { "ansi-styles": "^4.1.0", "supports-color": "^7.1.0" } }, "sha512-4D3B6Wf41KOYRFdszmDqMCGq5VV/uMAB273JILmO+3jAlh8X4qDtdtgCR3fxtbLEMzSx22QdhnDcJvu2u1fVwg=="],
"cliui/wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="],
"compress-commons/is-stream": ["is-stream@2.0.1", "", {}, "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg=="], "compress-commons/is-stream": ["is-stream@2.0.1", "", {}, "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg=="],
"decompress-response/mimic-response": ["mimic-response@3.1.0", "", {}, "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ=="], "decompress-response/mimic-response": ["mimic-response@3.1.0", "", {}, "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ=="],
@ -2550,14 +2568,14 @@
"execa/is-stream": ["is-stream@3.0.0", "", {}, "sha512-LnQR4bZ9IADDRSkvpqMGvt/tEJWclzklNgSw48V5EAaAeDd6qGvN8ei6k5p0tvxSR171VmGyHuTiAOfxAbr8kA=="], "execa/is-stream": ["is-stream@3.0.0", "", {}, "sha512-LnQR4bZ9IADDRSkvpqMGvt/tEJWclzklNgSw48V5EAaAeDd6qGvN8ei6k5p0tvxSR171VmGyHuTiAOfxAbr8kA=="],
"execa/signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
"express/cookie": ["cookie@0.7.2", "", {}, "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w=="], "express/cookie": ["cookie@0.7.2", "", {}, "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w=="],
"express/mime-types": ["mime-types@3.0.1", "", { "dependencies": { "mime-db": "^1.54.0" } }, "sha512-xRc4oEhT6eaBpU1XF7AjpOFD+xQmXNB5OVKwp4tqCuBpHLS/ZbBDrc07mYTDqVMg6PfxUjjNp85O6Cd2Z/5HWA=="], "express/mime-types": ["mime-types@3.0.1", "", { "dependencies": { "mime-db": "^1.54.0" } }, "sha512-xRc4oEhT6eaBpU1XF7AjpOFD+xQmXNB5OVKwp4tqCuBpHLS/ZbBDrc07mYTDqVMg6PfxUjjNp85O6Cd2Z/5HWA=="],
"fast-glob/glob-parent": ["glob-parent@5.1.2", "", { "dependencies": { "is-glob": "^4.0.1" } }, "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow=="], "fast-glob/glob-parent": ["glob-parent@5.1.2", "", { "dependencies": { "is-glob": "^4.0.1" } }, "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow=="],
"foreground-child/signal-exit": ["signal-exit@4.1.0", "", {}, "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw=="],
"get-uri/data-uri-to-buffer": ["data-uri-to-buffer@6.0.2", "", {}, "sha512-7hvf7/GW8e86rW0ptuwS3OcBGDjIi6SZva7hCyWC0yYry2cOPmLIjXAUHI6DK2HsnwJd9ifmt57i8eV2n4YNpw=="], "get-uri/data-uri-to-buffer": ["data-uri-to-buffer@6.0.2", "", {}, "sha512-7hvf7/GW8e86rW0ptuwS3OcBGDjIi6SZva7hCyWC0yYry2cOPmLIjXAUHI6DK2HsnwJd9ifmt57i8eV2n4YNpw=="],
"glob/minimatch": ["minimatch@9.0.5", "", { "dependencies": { "brace-expansion": "^2.0.1" } }, "sha512-G6T0ZX48xgozx7587koeX9Ys2NYy6Gmv//P89sEte9V9whIapMNF4idKxnW2QtCcLiTWlb/wfCabAtAFWhhBow=="], "glob/minimatch": ["minimatch@9.0.5", "", { "dependencies": { "brace-expansion": "^2.0.1" } }, "sha512-G6T0ZX48xgozx7587koeX9Ys2NYy6Gmv//P89sEte9V9whIapMNF4idKxnW2QtCcLiTWlb/wfCabAtAFWhhBow=="],
@ -2620,6 +2638,8 @@
"prebuild-install/tar-fs": ["tar-fs@2.1.3", "", { "dependencies": { "chownr": "^1.1.1", "mkdirp-classic": "^0.5.2", "pump": "^3.0.0", "tar-stream": "^2.1.4" } }, "sha512-090nwYJDmlhwFwEW3QQl+vaNnxsO2yVsd45eTKRBzSzu+hlb1w2K9inVq5b0ngXuLVqQ4ApvsUHHnu/zQNkWAg=="], "prebuild-install/tar-fs": ["tar-fs@2.1.3", "", { "dependencies": { "chownr": "^1.1.1", "mkdirp-classic": "^0.5.2", "pump": "^3.0.0", "tar-stream": "^2.1.4" } }, "sha512-090nwYJDmlhwFwEW3QQl+vaNnxsO2yVsd45eTKRBzSzu+hlb1w2K9inVq5b0ngXuLVqQ4ApvsUHHnu/zQNkWAg=="],
"proper-lockfile/signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
"protobufjs/@types/node": ["@types/node@22.15.32", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-3jigKqgSjsH6gYZv2nEsqdXfZqIFGAV36XYYjf9KGZ3PSG+IhLecqPnI310RvjutyMwifE2hhhNEklOUrvx/wA=="], "protobufjs/@types/node": ["@types/node@22.15.32", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-3jigKqgSjsH6gYZv2nEsqdXfZqIFGAV36XYYjf9KGZ3PSG+IhLecqPnI310RvjutyMwifE2hhhNEklOUrvx/wA=="],
"proxy-agent/lru-cache": ["lru-cache@7.18.3", "", {}, "sha512-jumlc0BIUrS3qJGgIkWZsyfAM7NCWiBcCDhnd+3NNM5KbBmLTgHVfWBcg6W+rLUsIpzpERPsvwUP7CckAQSOoA=="], "proxy-agent/lru-cache": ["lru-cache@7.18.3", "", {}, "sha512-jumlc0BIUrS3qJGgIkWZsyfAM7NCWiBcCDhnd+3NNM5KbBmLTgHVfWBcg6W+rLUsIpzpERPsvwUP7CckAQSOoA=="],
@ -2648,6 +2668,12 @@
"win-export-certificate-and-key/node-addon-api": ["node-addon-api@3.2.1", "", {}, "sha512-mmcei9JghVNDYydghQmeDX8KoAm0FAiYyIcUt/N4nhyAipB17pllZQDOJD2fotxABnt4Mdz+dKTO7eftLg4d0A=="], "win-export-certificate-and-key/node-addon-api": ["node-addon-api@3.2.1", "", {}, "sha512-mmcei9JghVNDYydghQmeDX8KoAm0FAiYyIcUt/N4nhyAipB17pllZQDOJD2fotxABnt4Mdz+dKTO7eftLg4d0A=="],
"wrap-ansi/ansi-styles": ["ansi-styles@6.2.1", "", {}, "sha512-bN798gFfQX+viw3R7yrGWRqnrN2oRkEkUjjl4JNn4E8GxxbjtG3FbrEIIY3l8/hrwUwIeCZvi4QuOTP4MErVug=="],
"wrap-ansi/string-width": ["string-width@5.1.2", "", { "dependencies": { "eastasianwidth": "^0.2.0", "emoji-regex": "^9.2.2", "strip-ansi": "^7.0.1" } }, "sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA=="],
"wrap-ansi/strip-ansi": ["strip-ansi@7.1.0", "", { "dependencies": { "ansi-regex": "^6.0.1" } }, "sha512-iq6eVVI64nQQTRYq2KtEg2d2uU7LElhTJwsH4YzIHZshxlgZms/wIc4VoDQTlG/IvVIrBKG06CrZnp0qv7hkcQ=="],
"yauzl/buffer-crc32": ["buffer-crc32@0.2.13", "", {}, "sha512-VO9Ht/+p3SN7SKWqcrgEzjGbRSJYTx+Q1pTQC0wrWqHx0vpJraQ6GtHx8tvcg1rlK1byhU5gccxgOgj7B0TDkQ=="], "yauzl/buffer-crc32": ["buffer-crc32@0.2.13", "", {}, "sha512-VO9Ht/+p3SN7SKWqcrgEzjGbRSJYTx+Q1pTQC0wrWqHx0vpJraQ6GtHx8tvcg1rlK1byhU5gccxgOgj7B0TDkQ=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="], "@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
@ -2660,8 +2686,6 @@
"@isaacs/cliui/strip-ansi/ansi-regex": ["ansi-regex@6.1.0", "", {}, "sha512-7HSX4QQb4CspciLpVFwyRe79O3xsIZDDLER21kERQ71oaPodF8jL725AgJMFAYbooIqolJoRLuM81SpeUkpkvA=="], "@isaacs/cliui/strip-ansi/ansi-regex": ["ansi-regex@6.1.0", "", {}, "sha512-7HSX4QQb4CspciLpVFwyRe79O3xsIZDDLER21kERQ71oaPodF8jL725AgJMFAYbooIqolJoRLuM81SpeUkpkvA=="],
"@isaacs/cliui/wrap-ansi/ansi-styles": ["ansi-styles@6.2.1", "", {}, "sha512-bN798gFfQX+viw3R7yrGWRqnrN2oRkEkUjjl4JNn4E8GxxbjtG3FbrEIIY3l8/hrwUwIeCZvi4QuOTP4MErVug=="],
"@mongodb-js/oidc-plugin/express/accepts": ["accepts@1.3.8", "", { "dependencies": { "mime-types": "~2.1.34", "negotiator": "0.6.3" } }, "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw=="], "@mongodb-js/oidc-plugin/express/accepts": ["accepts@1.3.8", "", { "dependencies": { "mime-types": "~2.1.34", "negotiator": "0.6.3" } }, "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw=="],
"@mongodb-js/oidc-plugin/express/body-parser": ["body-parser@1.20.3", "", { "dependencies": { "bytes": "3.1.2", "content-type": "~1.0.5", "debug": "2.6.9", "depd": "2.0.0", "destroy": "1.2.0", "http-errors": "2.0.0", "iconv-lite": "0.4.24", "on-finished": "2.4.1", "qs": "6.13.0", "raw-body": "2.5.2", "type-is": "~1.6.18", "unpipe": "1.0.0" } }, "sha512-7rAxByjUMqQ3/bHJy7D6OGXvx/MMc4IqBn/X0fcM1QUcAItpZrBEYhWGem+tzXH90c+G01ypMcYJBO9Y30203g=="], "@mongodb-js/oidc-plugin/express/body-parser": ["body-parser@1.20.3", "", { "dependencies": { "bytes": "3.1.2", "content-type": "~1.0.5", "debug": "2.6.9", "depd": "2.0.0", "destroy": "1.2.0", "http-errors": "2.0.0", "iconv-lite": "0.4.24", "on-finished": "2.4.1", "qs": "6.13.0", "raw-body": "2.5.2", "type-is": "~1.6.18", "unpipe": "1.0.0" } }, "sha512-7rAxByjUMqQ3/bHJy7D6OGXvx/MMc4IqBn/X0fcM1QUcAItpZrBEYhWGem+tzXH90c+G01ypMcYJBO9Y30203g=="],
@ -2880,12 +2904,18 @@
"run-applescript/execa/onetime": ["onetime@5.1.2", "", { "dependencies": { "mimic-fn": "^2.1.0" } }, "sha512-kbpaSSGJTWdAY5KPVeMOKXSrPtr8C8C7wodJbcsd51jRnmD+GZu8Y0VoU6Dm5Z4vWr0Ig/1NKuWRKf7j5aaYSg=="], "run-applescript/execa/onetime": ["onetime@5.1.2", "", { "dependencies": { "mimic-fn": "^2.1.0" } }, "sha512-kbpaSSGJTWdAY5KPVeMOKXSrPtr8C8C7wodJbcsd51jRnmD+GZu8Y0VoU6Dm5Z4vWr0Ig/1NKuWRKf7j5aaYSg=="],
"run-applescript/execa/signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
"run-applescript/execa/strip-final-newline": ["strip-final-newline@2.0.0", "", {}, "sha512-BrpvfNAE3dcvq7ll3xVumzjKjZQ5tI1sEUIKr3Uoks0XUl45St3FlatVqef9prk4jRDzhW6WZg+3bk93y6pLjA=="], "run-applescript/execa/strip-final-newline": ["strip-final-newline@2.0.0", "", {}, "sha512-BrpvfNAE3dcvq7ll3xVumzjKjZQ5tI1sEUIKr3Uoks0XUl45St3FlatVqef9prk4jRDzhW6WZg+3bk93y6pLjA=="],
"send/mime-types/mime-db": ["mime-db@1.54.0", "", {}, "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ=="], "send/mime-types/mime-db": ["mime-db@1.54.0", "", {}, "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ=="],
"type-is/mime-types/mime-db": ["mime-db@1.54.0", "", {}, "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ=="], "type-is/mime-types/mime-db": ["mime-db@1.54.0", "", {}, "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ=="],
"wrap-ansi/string-width/emoji-regex": ["emoji-regex@9.2.2", "", {}, "sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg=="],
"wrap-ansi/strip-ansi/ansi-regex": ["ansi-regex@6.1.0", "", {}, "sha512-7HSX4QQb4CspciLpVFwyRe79O3xsIZDDLER21kERQ71oaPodF8jL725AgJMFAYbooIqolJoRLuM81SpeUkpkvA=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="], "@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
"@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="], "@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],

2
knip.json
View file

@ -1,4 +1,4 @@
{ {
"entry": ["src/index.ts"], "entry": ["src/index.ts"],
"project": ["src/**/*.ts"] "project": ["src/**/*.ts"]
} }

46
libs/core/cache/src/cache-factory.ts vendored
View file

@ -1,23 +1,23 @@
import { NamespacedCache } from './namespaced-cache'; import { NamespacedCache } from './namespaced-cache';
import type { CacheProvider } from './types'; import type { CacheProvider } from './types';
/** /**
* Factory function to create namespaced caches * Factory function to create namespaced caches
* Provides a clean API for services to get their own namespaced cache * Provides a clean API for services to get their own namespaced cache
*/ */
export function createNamespacedCache( export function createNamespacedCache(
cache: CacheProvider | null | undefined, cache: CacheProvider | null | undefined,
namespace: string namespace: string
): CacheProvider | null { ): CacheProvider | null {
if (!cache) { if (!cache) {
return null; return null;
} }
return new NamespacedCache(cache, namespace); return new NamespacedCache(cache, namespace);
} }
/** /**
* Type guard to check if cache is available * Type guard to check if cache is available
*/ */
export function isCacheAvailable(cache: any): cache is CacheProvider { export function isCacheAvailable(cache: any): cache is CacheProvider {
return cache !== null && cache !== undefined && typeof cache.get === 'function'; return cache !== null && cache !== undefined && typeof cache.get === 'function';
} }

2
libs/core/cache/src/connection-manager.ts vendored
View file

@ -88,7 +88,7 @@ export class RedisConnectionManager {
}; };
const redis = new Redis(redisOptions); const redis = new Redis(redisOptions);
// Use the provided logger or fall back to instance logger // Use the provided logger or fall back to instance logger
const log = logger || this.logger; const log = logger || this.logger;

201
libs/core/cache/src/namespaced-cache.ts vendored
View file

@ -1,101 +1,100 @@
import type { CacheProvider } from './types'; import type { CacheProvider } from './types';
/** /**
* A cache wrapper that automatically prefixes all keys with a namespace * A cache wrapper that automatically prefixes all keys with a namespace
* Used to provide isolated cache spaces for different services * Used to provide isolated cache spaces for different services
*/ */
export class NamespacedCache implements CacheProvider { export class NamespacedCache implements CacheProvider {
private readonly prefix: string; private readonly prefix: string;
constructor( constructor(
private readonly cache: CacheProvider, private readonly cache: CacheProvider,
private readonly namespace: string private readonly namespace: string
) { ) {
this.prefix = `cache:${namespace}:`; this.prefix = `cache:${namespace}:`;
} }
async get<T = any>(key: string): Promise<T | null> { async get<T = any>(key: string): Promise<T | null> {
return this.cache.get(`${this.prefix}${key}`); return this.cache.get(`${this.prefix}${key}`);
} }
async set<T>( async set<T>(
key: string, key: string,
value: T, value: T,
options?: options?:
| number | number
| { | {
ttl?: number; ttl?: number;
preserveTTL?: boolean; preserveTTL?: boolean;
onlyIfExists?: boolean; onlyIfExists?: boolean;
onlyIfNotExists?: boolean; onlyIfNotExists?: boolean;
getOldValue?: boolean; getOldValue?: boolean;
} }
): Promise<T | null> { ): Promise<T | null> {
return this.cache.set(`${this.prefix}${key}`, value, options); return this.cache.set(`${this.prefix}${key}`, value, options);
} }
async del(key: string): Promise<void> { async del(key: string): Promise<void> {
return this.cache.del(`${this.prefix}${key}`); return this.cache.del(`${this.prefix}${key}`);
} }
async exists(key: string): Promise<boolean> { async exists(key: string): Promise<boolean> {
return this.cache.exists(`${this.prefix}${key}`); return this.cache.exists(`${this.prefix}${key}`);
} }
async keys(pattern: string = '*'): Promise<string[]> { async keys(pattern: string = '*'): Promise<string[]> {
const fullPattern = `${this.prefix}${pattern}`; const fullPattern = `${this.prefix}${pattern}`;
const keys = await this.cache.keys(fullPattern); const keys = await this.cache.keys(fullPattern);
// Remove the prefix from returned keys for cleaner API // Remove the prefix from returned keys for cleaner API
return keys.map(k => k.substring(this.prefix.length)); return keys.map(k => k.substring(this.prefix.length));
} }
async clear(): Promise<void> { async clear(): Promise<void> {
// Clear only keys with this namespace prefix // Clear only keys with this namespace prefix
const keys = await this.cache.keys(`${this.prefix}*`); const keys = await this.cache.keys(`${this.prefix}*`);
if (keys.length > 0) { if (keys.length > 0) {
await Promise.all(keys.map(key => this.cache.del(key))); await Promise.all(keys.map(key => this.cache.del(key)));
} }
} }
getStats() {
getStats() { return this.cache.getStats();
return this.cache.getStats(); }
}
async health(): Promise<boolean> {
async health(): Promise<boolean> { return this.cache.health();
return this.cache.health(); }
}
isReady(): boolean {
isReady(): boolean { return this.cache.isReady();
return this.cache.isReady(); }
}
async waitForReady(timeout?: number): Promise<void> {
async waitForReady(timeout?: number): Promise<void> { return this.cache.waitForReady(timeout);
return this.cache.waitForReady(timeout); }
}
async close(): Promise<void> {
async close(): Promise<void> { // Namespaced cache doesn't own the connection, so we don't close it
// Namespaced cache doesn't own the connection, so we don't close it // The underlying cache instance should be closed by its owner
// The underlying cache instance should be closed by its owner }
}
getNamespace(): string {
getNamespace(): string { return this.namespace;
return this.namespace; }
}
getFullPrefix(): string {
getFullPrefix(): string { return this.prefix;
return this.prefix; }
}
/**
/** * Get a value using a raw Redis key (bypassing the namespace prefix)
* Get a value using a raw Redis key (bypassing the namespace prefix) * Delegates to the underlying cache's getRaw method if available
* Delegates to the underlying cache's getRaw method if available */
*/ async getRaw<T = unknown>(key: string): Promise<T | null> {
async getRaw<T = unknown>(key: string): Promise<T | null> { if (this.cache.getRaw) {
if (this.cache.getRaw) { return this.cache.getRaw<T>(key);
return this.cache.getRaw<T>(key); }
} // Fallback for caches that don't implement getRaw
// Fallback for caches that don't implement getRaw return null;
return null; }
} }
}

6
libs/core/config/src/config-manager.ts
View file

@ -1,9 +1,9 @@
import { join } from 'path'; import { join } from 'path';
import { z } from 'zod'; import { z } from 'zod';
import { getLogger } from '@stock-bot/logger';
import { EnvLoader } from './loaders/env.loader'; import { EnvLoader } from './loaders/env.loader';
import { FileLoader } from './loaders/file.loader'; import { FileLoader } from './loaders/file.loader';
import { ConfigError, ConfigValidationError } from './errors'; import { ConfigError, ConfigValidationError } from './errors';
import { getLogger } from '@stock-bot/logger';
import type { import type {
ConfigLoader, ConfigLoader,
ConfigManagerOptions, ConfigManagerOptions,
@ -82,9 +82,9 @@ export class ConfigManager<T = Record<string, unknown>> {
expected: (err as any).expected, expected: (err as any).expected,
received: (err as any).received, received: (err as any).received,
})); }));
this.logger.error('Configuration validation failed:', errorDetails); this.logger.error('Configuration validation failed:', errorDetails);
throw new ConfigValidationError('Configuration validation failed', error.errors); throw new ConfigValidationError('Configuration validation failed', error.errors);
} }
throw error; throw error;

5
libs/core/config/src/index.ts
View file

@ -1,10 +1,10 @@
// Import necessary types // Import necessary types
import { z } from 'zod';
import { EnvLoader } from './loaders/env.loader'; import { EnvLoader } from './loaders/env.loader';
import { FileLoader } from './loaders/file.loader'; import { FileLoader } from './loaders/file.loader';
import { ConfigManager } from './config-manager'; import { ConfigManager } from './config-manager';
import type { BaseAppConfig } from './schemas'; import type { BaseAppConfig } from './schemas';
import { baseAppSchema } from './schemas'; import { baseAppSchema } from './schemas';
import { z } from 'zod';
// Legacy singleton instance for backward compatibility // Legacy singleton instance for backward compatibility
let configInstance: ConfigManager<BaseAppConfig> | null = null; let configInstance: ConfigManager<BaseAppConfig> | null = null;
@ -56,7 +56,6 @@ function loadCriticalEnvVarsSync(): void {
// Load critical env vars immediately // Load critical env vars immediately
loadCriticalEnvVarsSync(); loadCriticalEnvVarsSync();
/** /**
* Initialize configuration for a service in a monorepo. * Initialize configuration for a service in a monorepo.
* Automatically loads configs from: * Automatically loads configs from:
@ -121,8 +120,6 @@ export function getLogConfig() {
return getConfig().log; return getConfig().log;
} }
export function getQueueConfig() { export function getQueueConfig() {
return getConfig().queue; return getConfig().queue;
} }

10
libs/core/config/src/schemas/__tests__/unified-app.test.ts
View file

@ -1,5 +1,5 @@
import { describe, expect, it } from 'bun:test'; import { describe, expect, it } from 'bun:test';
import { unifiedAppSchema, toUnifiedConfig, getStandardServiceName } from '../unified-app.schema'; import { getStandardServiceName, toUnifiedConfig, unifiedAppSchema } from '../unified-app.schema';
describe('UnifiedAppConfig', () => { describe('UnifiedAppConfig', () => {
describe('getStandardServiceName', () => { describe('getStandardServiceName', () => {
@ -74,13 +74,13 @@ describe('UnifiedAppConfig', () => {
}; };
const result = unifiedAppSchema.parse(config); const result = unifiedAppSchema.parse(config);
// Should have both nested and flat structure // Should have both nested and flat structure
expect(result.postgres).toBeDefined(); expect(result.postgres).toBeDefined();
expect(result.mongodb).toBeDefined(); expect(result.mongodb).toBeDefined();
expect(result.database?.postgres).toBeDefined(); expect(result.database?.postgres).toBeDefined();
expect(result.database?.mongodb).toBeDefined(); expect(result.database?.mongodb).toBeDefined();
// Values should match // Values should match
expect(result.postgres?.host).toBe('localhost'); expect(result.postgres?.host).toBe('localhost');
expect(result.postgres?.port).toBe(5432); expect(result.postgres?.port).toBe(5432);
@ -144,7 +144,7 @@ describe('UnifiedAppConfig', () => {
}; };
const unified = toUnifiedConfig(stockBotConfig); const unified = toUnifiedConfig(stockBotConfig);
expect(unified.service.serviceName).toBe('data-ingestion'); expect(unified.service.serviceName).toBe('data-ingestion');
expect(unified.redis).toBeDefined(); expect(unified.redis).toBeDefined();
expect(unified.redis?.host).toBe('localhost'); expect(unified.redis?.host).toBe('localhost');
@ -152,4 +152,4 @@ describe('UnifiedAppConfig', () => {
expect(unified.postgres?.host).toBe('localhost'); expect(unified.postgres?.host).toBe('localhost');
}); });
}); });
}); });

124
libs/core/config/src/schemas/base-app.schema.ts
View file

@ -1,61 +1,63 @@
import { z } from 'zod'; import { z } from 'zod';
import { environmentSchema } from './base.schema'; import { environmentSchema } from './base.schema';
import { import {
postgresConfigSchema, dragonflyConfigSchema,
mongodbConfigSchema, mongodbConfigSchema,
questdbConfigSchema, postgresConfigSchema,
dragonflyConfigSchema questdbConfigSchema,
} from './database.schema'; } from './database.schema';
import { import {
serviceConfigSchema, browserConfigSchema,
loggingConfigSchema, httpConfigSchema,
queueConfigSchema, loggingConfigSchema,
httpConfigSchema, proxyConfigSchema,
webshareConfigSchema, queueConfigSchema,
browserConfigSchema, serviceConfigSchema,
proxyConfigSchema webshareConfigSchema,
} from './service.schema'; } from './service.schema';
/** /**
* Generic base application schema that can be extended by specific apps * Generic base application schema that can be extended by specific apps
*/ */
export const baseAppSchema = z.object({ export const baseAppSchema = z.object({
// Basic app info // Basic app info
name: z.string(), name: z.string(),
version: z.string(), version: z.string(),
environment: environmentSchema.default('development'), environment: environmentSchema.default('development'),
// Service configuration // Service configuration
service: serviceConfigSchema, service: serviceConfigSchema,
// Logging configuration // Logging configuration
log: loggingConfigSchema, log: loggingConfigSchema,
// Database configuration - apps can choose which databases they need // Database configuration - apps can choose which databases they need
database: z.object({ database: z
postgres: postgresConfigSchema.optional(), .object({
mongodb: mongodbConfigSchema.optional(), postgres: postgresConfigSchema.optional(),
questdb: questdbConfigSchema.optional(), mongodb: mongodbConfigSchema.optional(),
dragonfly: dragonflyConfigSchema.optional(), questdb: questdbConfigSchema.optional(),
}).optional(), dragonfly: dragonflyConfigSchema.optional(),
})
// Redis configuration (used for cache and queue) .optional(),
redis: dragonflyConfigSchema.optional(),
// Redis configuration (used for cache and queue)
// Queue configuration redis: dragonflyConfigSchema.optional(),
queue: queueConfigSchema.optional(),
// Queue configuration
// HTTP client configuration queue: queueConfigSchema.optional(),
http: httpConfigSchema.optional(),
// HTTP client configuration
// WebShare proxy configuration http: httpConfigSchema.optional(),
webshare: webshareConfigSchema.optional(),
// WebShare proxy configuration
// Browser configuration webshare: webshareConfigSchema.optional(),
browser: browserConfigSchema.optional(),
// Browser configuration
// Proxy manager configuration browser: browserConfigSchema.optional(),
proxy: proxyConfigSchema.optional(),
}); // Proxy manager configuration
proxy: proxyConfigSchema.optional(),
export type BaseAppConfig = z.infer<typeof baseAppSchema>; });
export type BaseAppConfig = z.infer<typeof baseAppSchema>;

1
libs/core/config/src/schemas/index.ts
View file

@ -15,4 +15,3 @@ export type { BaseAppConfig } from './base-app.schema';
// Export unified schema for standardized configuration // Export unified schema for standardized configuration
export { unifiedAppSchema, toUnifiedConfig, getStandardServiceName } from './unified-app.schema'; export { unifiedAppSchema, toUnifiedConfig, getStandardServiceName } from './unified-app.schema';
export type { UnifiedAppConfig } from './unified-app.schema'; export type { UnifiedAppConfig } from './unified-app.schema';

10
libs/core/config/src/schemas/service.schema.ts
View file

@ -100,8 +100,10 @@ export const proxyConfigSchema = z.object({
enabled: z.boolean().default(false), enabled: z.boolean().default(false),
cachePrefix: z.string().default('proxy:'), cachePrefix: z.string().default('proxy:'),
ttl: z.number().default(3600), ttl: z.number().default(3600),
webshare: z.object({ webshare: z
apiKey: z.string(), .object({
apiUrl: z.string().default('https://proxy.webshare.io/api/v2/'), apiKey: z.string(),
}).optional(), apiUrl: z.string().default('https://proxy.webshare.io/api/v2/'),
})
.optional(),
}); });

104
libs/core/config/src/schemas/unified-app.schema.ts
View file

@ -1,62 +1,67 @@
import { z } from 'zod'; import { z } from 'zod';
import { baseAppSchema } from './base-app.schema'; import { baseAppSchema } from './base-app.schema';
import { import {
postgresConfigSchema, dragonflyConfigSchema,
mongodbConfigSchema, mongodbConfigSchema,
questdbConfigSchema, postgresConfigSchema,
dragonflyConfigSchema questdbConfigSchema,
} from './database.schema'; } from './database.schema';
/** /**
* Unified application configuration schema that provides both nested and flat access * Unified application configuration schema that provides both nested and flat access
* to database configurations for backward compatibility while maintaining a clean structure * to database configurations for backward compatibility while maintaining a clean structure
*/ */
export const unifiedAppSchema = baseAppSchema.extend({ export const unifiedAppSchema = baseAppSchema
// Flat database configs for DI system (these take precedence) .extend({
redis: dragonflyConfigSchema.optional(), // Flat database configs for DI system (these take precedence)
mongodb: mongodbConfigSchema.optional(), redis: dragonflyConfigSchema.optional(),
postgres: postgresConfigSchema.optional(), mongodb: mongodbConfigSchema.optional(),
questdb: questdbConfigSchema.optional(), postgres: postgresConfigSchema.optional(),
}).transform((data) => { questdb: questdbConfigSchema.optional(),
// Ensure service.serviceName is set from service.name if not provided })
if (data.service && !data.service.serviceName) { .transform(data => {
data.service.serviceName = data.service.name.replace(/([A-Z])/g, '-$1').toLowerCase().replace(/^-/, ''); // Ensure service.serviceName is set from service.name if not provided
} if (data.service && !data.service.serviceName) {
data.service.serviceName = data.service.name
.replace(/([A-Z])/g, '-$1')
.toLowerCase()
.replace(/^-/, '');
}
// If flat configs exist, ensure they're also in the nested database object // If flat configs exist, ensure they're also in the nested database object
if (data.redis || data.mongodb || data.postgres || data.questdb) { if (data.redis || data.mongodb || data.postgres || data.questdb) {
data.database = { data.database = {
...data.database, ...data.database,
dragonfly: data.redis || data.database?.dragonfly, dragonfly: data.redis || data.database?.dragonfly,
mongodb: data.mongodb || data.database?.mongodb, mongodb: data.mongodb || data.database?.mongodb,
postgres: data.postgres || data.database?.postgres, postgres: data.postgres || data.database?.postgres,
questdb: data.questdb || data.database?.questdb, questdb: data.questdb || data.database?.questdb,
}; };
} }
// If nested configs exist but flat ones don't, copy them to flat structure // If nested configs exist but flat ones don't, copy them to flat structure
if (data.database) { if (data.database) {
if (data.database.dragonfly && !data.redis) { if (data.database.dragonfly && !data.redis) {
data.redis = data.database.dragonfly; data.redis = data.database.dragonfly;
} }
if (data.database.mongodb && !data.mongodb) { if (data.database.mongodb && !data.mongodb) {
data.mongodb = data.database.mongodb; data.mongodb = data.database.mongodb;
} }
if (data.database.postgres && !data.postgres) { if (data.database.postgres && !data.postgres) {
data.postgres = data.database.postgres; data.postgres = data.database.postgres;
} }
if (data.database.questdb && !data.questdb) { if (data.database.questdb && !data.questdb) {
// Handle the ilpPort -> influxPort mapping for DI system // Handle the ilpPort -> influxPort mapping for DI system
const questdbConfig = { ...data.database.questdb }; const questdbConfig = { ...data.database.questdb };
if ('ilpPort' in questdbConfig && !('influxPort' in questdbConfig)) { if ('ilpPort' in questdbConfig && !('influxPort' in questdbConfig)) {
(questdbConfig as any).influxPort = questdbConfig.ilpPort; (questdbConfig as any).influxPort = questdbConfig.ilpPort;
}
data.questdb = questdbConfig;
} }
data.questdb = questdbConfig;
} }
}
return data; return data;
}); });
export type UnifiedAppConfig = z.infer<typeof unifiedAppSchema>; export type UnifiedAppConfig = z.infer<typeof unifiedAppSchema>;
@ -72,5 +77,8 @@ export function toUnifiedConfig(config: any): UnifiedAppConfig {
*/ */
export function getStandardServiceName(serviceName: string): string { export function getStandardServiceName(serviceName: string): string {
// Convert camelCase to kebab-case // Convert camelCase to kebab-case
return serviceName.replace(/([A-Z])/g, '-$1').toLowerCase().replace(/^-/, ''); return serviceName
} .replace(/([A-Z])/g, '-$1')
.toLowerCase()
.replace(/^-/, '');
}

2
libs/core/di/package.json
View file

@ -20,6 +20,8 @@
"@stock-bot/queue": "workspace:*", "@stock-bot/queue": "workspace:*",
"@stock-bot/shutdown": "workspace:*", "@stock-bot/shutdown": "workspace:*",
"@stock-bot/handlers": "workspace:*", "@stock-bot/handlers": "workspace:*",
"@stock-bot/handler-registry": "workspace:*",
"glob": "^10.0.0",
"zod": "^3.23.8", "zod": "^3.23.8",
"hono": "^4.0.0", "hono": "^4.0.0",
"awilix": "^12.0.5" "awilix": "^12.0.5"

8
libs/core/di/src/awilix-container.ts
View file

@ -3,16 +3,16 @@
* Creates a decoupled, reusable dependency injection container * Creates a decoupled, reusable dependency injection container
*/ */
import { type AwilixContainer } from 'awilix';
import type { Browser } from '@stock-bot/browser'; import type { Browser } from '@stock-bot/browser';
import type { CacheProvider } from '@stock-bot/cache'; import type { CacheProvider } from '@stock-bot/cache';
import type { IServiceContainer } from '@stock-bot/types';
import type { Logger } from '@stock-bot/logger'; import type { Logger } from '@stock-bot/logger';
import type { MongoDBClient } from '@stock-bot/mongodb'; import type { MongoDBClient } from '@stock-bot/mongodb';
import type { PostgreSQLClient } from '@stock-bot/postgres'; import type { PostgreSQLClient } from '@stock-bot/postgres';
import type { ProxyManager } from '@stock-bot/proxy'; import type { ProxyManager } from '@stock-bot/proxy';
import type { QuestDBClient } from '@stock-bot/questdb'; import type { QuestDBClient } from '@stock-bot/questdb';
import type { QueueManager } from '@stock-bot/queue'; import type { QueueManager } from '@stock-bot/queue';
import { type AwilixContainer } from 'awilix'; import type { IServiceContainer } from '@stock-bot/types';
import type { AppConfig } from './config/schemas'; import type { AppConfig } from './config/schemas';
// Re-export for backward compatibility // Re-export for backward compatibility
@ -41,8 +41,6 @@ export interface ServiceDefinitions {
serviceContainer: IServiceContainer; serviceContainer: IServiceContainer;
} }
// Export typed container // Export typed container
export type ServiceContainer = AwilixContainer<ServiceDefinitions>; export type ServiceContainer = AwilixContainer<ServiceDefinitions>;
export type ServiceCradle = ServiceDefinitions; export type ServiceCradle = ServiceDefinitions;
@ -59,5 +57,3 @@ export interface ServiceContainerOptions {
enableBrowser?: boolean; enableBrowser?: boolean;
enableProxy?: boolean; enableProxy?: boolean;
} }

18
libs/core/di/src/config/schemas/index.ts
View file

@ -1,9 +1,9 @@
import { z } from 'zod'; import { z } from 'zod';
import { redisConfigSchema } from './redis.schema';
import { mongodbConfigSchema } from './mongodb.schema'; import { mongodbConfigSchema } from './mongodb.schema';
import { postgresConfigSchema } from './postgres.schema'; import { postgresConfigSchema } from './postgres.schema';
import { questdbConfigSchema } from './questdb.schema'; import { questdbConfigSchema } from './questdb.schema';
import { proxyConfigSchema, browserConfigSchema, queueConfigSchema } from './service.schema'; import { redisConfigSchema } from './redis.schema';
import { browserConfigSchema, proxyConfigSchema, queueConfigSchema } from './service.schema';
export const appConfigSchema = z.object({ export const appConfigSchema = z.object({
redis: redisConfigSchema, redis: redisConfigSchema,
@ -13,11 +13,13 @@ export const appConfigSchema = z.object({
proxy: proxyConfigSchema.optional(), proxy: proxyConfigSchema.optional(),
browser: browserConfigSchema.optional(), browser: browserConfigSchema.optional(),
queue: queueConfigSchema.optional(), queue: queueConfigSchema.optional(),
service: z.object({ service: z
name: z.string(), .object({
serviceName: z.string().optional(), // Standard kebab-case service name name: z.string(),
port: z.number().optional(), serviceName: z.string().optional(), // Standard kebab-case service name
}).optional(), port: z.number().optional(),
})
.optional(),
}); });
export type AppConfig = z.infer<typeof appConfigSchema>; export type AppConfig = z.infer<typeof appConfigSchema>;
@ -27,4 +29,4 @@ export * from './redis.schema';
export * from './mongodb.schema'; export * from './mongodb.schema';
export * from './postgres.schema'; export * from './postgres.schema';
export * from './questdb.schema'; export * from './questdb.schema';
export * from './service.schema'; export * from './service.schema';

18
libs/core/di/src/config/schemas/mongodb.schema.ts
View file

@ -1,9 +1,9 @@
import { z } from 'zod'; import { z } from 'zod';
export const mongodbConfigSchema = z.object({ export const mongodbConfigSchema = z.object({
enabled: z.boolean().optional().default(true), enabled: z.boolean().optional().default(true),
uri: z.string(), uri: z.string(),
database: z.string(), database: z.string(),
}); });
export type MongoDBConfig = z.infer<typeof mongodbConfigSchema>; export type MongoDBConfig = z.infer<typeof mongodbConfigSchema>;

24
libs/core/di/src/config/schemas/postgres.schema.ts
View file

@ -1,12 +1,12 @@
import { z } from 'zod'; import { z } from 'zod';
export const postgresConfigSchema = z.object({ export const postgresConfigSchema = z.object({
enabled: z.boolean().optional().default(true), enabled: z.boolean().optional().default(true),
host: z.string().default('localhost'), host: z.string().default('localhost'),
port: z.number().default(5432), port: z.number().default(5432),
database: z.string(), database: z.string(),
user: z.string(), user: z.string(),
password: z.string(), password: z.string(),
}); });
export type PostgresConfig = z.infer<typeof postgresConfigSchema>; export type PostgresConfig = z.infer<typeof postgresConfigSchema>;

24
libs/core/di/src/config/schemas/questdb.schema.ts
View file

@ -1,12 +1,12 @@
import { z } from 'zod'; import { z } from 'zod';
export const questdbConfigSchema = z.object({ export const questdbConfigSchema = z.object({
enabled: z.boolean().optional().default(true), enabled: z.boolean().optional().default(true),
host: z.string().default('localhost'), host: z.string().default('localhost'),
httpPort: z.number().optional().default(9000), httpPort: z.number().optional().default(9000),
pgPort: z.number().optional().default(8812), pgPort: z.number().optional().default(8812),
influxPort: z.number().optional().default(9009), influxPort: z.number().optional().default(9009),
database: z.string().optional().default('questdb'), database: z.string().optional().default('questdb'),
}); });
export type QuestDBConfig = z.infer<typeof questdbConfigSchema>; export type QuestDBConfig = z.infer<typeof questdbConfigSchema>;

24
libs/core/di/src/config/schemas/redis.schema.ts
View file

@ -1,12 +1,12 @@
import { z } from 'zod'; import { z } from 'zod';
export const redisConfigSchema = z.object({ export const redisConfigSchema = z.object({
enabled: z.boolean().optional().default(true), enabled: z.boolean().optional().default(true),
host: z.string().default('localhost'), host: z.string().default('localhost'),
port: z.number().default(6379), port: z.number().default(6379),
password: z.string().optional(), password: z.string().optional(),
username: z.string().optional(), username: z.string().optional(),
db: z.number().optional().default(0), db: z.number().optional().default(0),
}); });
export type RedisConfig = z.infer<typeof redisConfigSchema>; export type RedisConfig = z.infer<typeof redisConfigSchema>;

37
libs/core/di/src/config/schemas/service.schema.ts
View file

@ -4,10 +4,12 @@ export const proxyConfigSchema = z.object({
enabled: z.boolean().default(false), enabled: z.boolean().default(false),
cachePrefix: z.string().optional().default('proxy:'), cachePrefix: z.string().optional().default('proxy:'),
ttl: z.number().optional().default(3600), ttl: z.number().optional().default(3600),
webshare: z.object({ webshare: z
apiKey: z.string(), .object({
apiUrl: z.string().default('https://proxy.webshare.io/api/v2/'), apiKey: z.string(),
}).optional(), apiUrl: z.string().default('https://proxy.webshare.io/api/v2/'),
})
.optional(),
}); });
export const browserConfigSchema = z.object({ export const browserConfigSchema = z.object({
@ -21,18 +23,23 @@ export const queueConfigSchema = z.object({
concurrency: z.number().optional().default(1), concurrency: z.number().optional().default(1),
enableScheduledJobs: z.boolean().optional().default(true), enableScheduledJobs: z.boolean().optional().default(true),
delayWorkerStart: z.boolean().optional().default(false), delayWorkerStart: z.boolean().optional().default(false),
defaultJobOptions: z.object({ defaultJobOptions: z
attempts: z.number().default(3), .object({
backoff: z.object({ attempts: z.number().default(3),
type: z.enum(['exponential', 'fixed']).default('exponential'), backoff: z
delay: z.number().default(1000), .object({
}).default({}), type: z.enum(['exponential', 'fixed']).default('exponential'),
removeOnComplete: z.number().default(100), delay: z.number().default(1000),
removeOnFail: z.number().default(50), })
timeout: z.number().optional(), .default({}),
}).optional().default({}), removeOnComplete: z.number().default(100),
removeOnFail: z.number().default(50),
timeout: z.number().optional(),
})
.optional()
.default({}),
}); });
export type ProxyConfig = z.infer<typeof proxyConfigSchema>; export type ProxyConfig = z.infer<typeof proxyConfigSchema>;
export type BrowserConfig = z.infer<typeof browserConfigSchema>; export type BrowserConfig = z.infer<typeof browserConfigSchema>;
export type QueueConfig = z.infer<typeof queueConfigSchema>; export type QueueConfig = z.infer<typeof queueConfigSchema>;

152
libs/core/di/src/container/builder.ts
View file

@ -1,15 +1,17 @@
import { createContainer, InjectionMode, asFunction, type AwilixContainer } from 'awilix'; import { asClass, asFunction, createContainer, InjectionMode, type AwilixContainer } from 'awilix';
import type { BaseAppConfig as StockBotAppConfig, UnifiedAppConfig } from '@stock-bot/config'; import type { BaseAppConfig as StockBotAppConfig, UnifiedAppConfig } from '@stock-bot/config';
import { appConfigSchema, type AppConfig } from '../config/schemas';
import { toUnifiedConfig } from '@stock-bot/config'; import { toUnifiedConfig } from '@stock-bot/config';
import { import { HandlerRegistry } from '@stock-bot/handler-registry';
registerCoreServices, import { appConfigSchema, type AppConfig } from '../config/schemas';
import {
registerApplicationServices,
registerCacheServices, registerCacheServices,
registerCoreServices,
registerDatabaseServices, registerDatabaseServices,
registerApplicationServices
} from '../registrations'; } from '../registrations';
import { HandlerScanner } from '../scanner';
import { ServiceLifecycleManager } from '../utils/lifecycle'; import { ServiceLifecycleManager } from '../utils/lifecycle';
import type { ServiceDefinitions, ContainerBuildOptions } from './types'; import type { ContainerBuildOptions, ServiceDefinitions } from './types';
export class ServiceContainerBuilder { export class ServiceContainerBuilder {
private config: Partial<AppConfig> = {}; private config: Partial<AppConfig> = {};
@ -38,7 +40,10 @@ export class ServiceContainerBuilder {
return this; return this;
} }
enableService(service: keyof Omit<ContainerBuildOptions, 'skipInitialization' | 'initializationTimeout'>, enabled = true): this { enableService(
service: keyof Omit<ContainerBuildOptions, 'skipInitialization' | 'initializationTimeout'>,
enabled = true
): this {
this.options[service] = enabled; this.options[service] = enabled;
return this; return this;
} }
@ -51,7 +56,7 @@ export class ServiceContainerBuilder {
async build(): Promise<AwilixContainer<ServiceDefinitions>> { async build(): Promise<AwilixContainer<ServiceDefinitions>> {
// Validate and prepare config // Validate and prepare config
const validatedConfig = this.prepareConfig(); const validatedConfig = this.prepareConfig();
// Create container // Create container
const container = createContainer<ServiceDefinitions>({ const container = createContainer<ServiceDefinitions>({
injectionMode: InjectionMode.PROXY, injectionMode: InjectionMode.PROXY,
@ -77,17 +82,19 @@ export class ServiceContainerBuilder {
private applyServiceOptions(config: Partial<AppConfig>): AppConfig { private applyServiceOptions(config: Partial<AppConfig>): AppConfig {
// Ensure questdb config has the right field names for DI // Ensure questdb config has the right field names for DI
const questdbConfig = config.questdb ? { const questdbConfig = config.questdb
...config.questdb, ? {
influxPort: (config.questdb as any).influxPort || (config.questdb as any).ilpPort || 9009, ...config.questdb,
} : { influxPort: (config.questdb as any).influxPort || (config.questdb as any).ilpPort || 9009,
enabled: true, }
host: 'localhost', : {
httpPort: 9000, enabled: true,
pgPort: 8812, host: 'localhost',
influxPort: 9009, httpPort: 9000,
database: 'questdb', pgPort: 8812,
}; influxPort: 9009,
database: 'questdb',
};
return { return {
redis: config.redis || { redis: config.redis || {
@ -110,61 +117,88 @@ export class ServiceContainerBuilder {
password: 'postgres', password: 'postgres',
}, },
questdb: this.options.enableQuestDB ? questdbConfig : undefined, questdb: this.options.enableQuestDB ? questdbConfig : undefined,
proxy: this.options.enableProxy ? (config.proxy || { enabled: false, cachePrefix: 'proxy:', ttl: 3600 }) : undefined, proxy: this.options.enableProxy
browser: this.options.enableBrowser ? (config.browser || { headless: true, timeout: 30000 }) : undefined, ? config.proxy || { enabled: false, cachePrefix: 'proxy:', ttl: 3600 }
queue: this.options.enableQueue ? (config.queue || { : undefined,
enabled: true, browser: this.options.enableBrowser
workers: 1, ? config.browser || { headless: true, timeout: 30000 }
concurrency: 1, : undefined,
enableScheduledJobs: true, queue: this.options.enableQueue
delayWorkerStart: false, ? config.queue || {
defaultJobOptions: { enabled: true,
attempts: 3, workers: 1,
backoff: { type: 'exponential' as const, delay: 1000 }, concurrency: 1,
removeOnComplete: 100, enableScheduledJobs: true,
removeOnFail: 50, delayWorkerStart: false,
} defaultJobOptions: {
}) : undefined, attempts: 3,
backoff: { type: 'exponential' as const, delay: 1000 },
removeOnComplete: 100,
removeOnFail: 50,
},
}
: undefined,
service: config.service, service: config.service,
}; };
} }
private registerServices(container: AwilixContainer<ServiceDefinitions>, config: AppConfig): void { private registerServices(
container: AwilixContainer<ServiceDefinitions>,
config: AppConfig
): void {
// Register handler infrastructure first
container.register({
handlerRegistry: asClass(HandlerRegistry).singleton(),
handlerScanner: asClass(HandlerScanner).singleton(),
});
registerCoreServices(container, config); registerCoreServices(container, config);
registerCacheServices(container, config); registerCacheServices(container, config);
registerDatabaseServices(container, config); registerDatabaseServices(container, config);
registerApplicationServices(container, config); registerApplicationServices(container, config);
// Register service container aggregate // Register service container aggregate
container.register({ container.register({
serviceContainer: asFunction(({ serviceContainer: asFunction(
config: _config, logger, cache, globalCache, proxyManager, browser, ({
queueManager, mongoClient, postgresClient, questdbClient config: _config,
}) => ({ logger,
logger, cache,
cache, globalCache,
globalCache, proxyManager,
proxy: proxyManager, // Map proxyManager to proxy browser,
browser, queueManager,
queue: queueManager, // Map queueManager to queue mongoClient,
mongodb: mongoClient, // Map mongoClient to mongodb postgresClient,
postgres: postgresClient, // Map postgresClient to postgres questdbClient,
questdb: questdbClient, // Map questdbClient to questdb }) => ({
})).singleton(), logger,
cache,
globalCache,
proxy: proxyManager, // Map proxyManager to proxy
browser,
queue: queueManager, // Map queueManager to queue
mongodb: mongoClient, // Map mongoClient to mongodb
postgres: postgresClient, // Map postgresClient to postgres
questdb: questdbClient, // Map questdbClient to questdb
})
).singleton(),
}); });
} }
private transformStockBotConfig(config: UnifiedAppConfig): Partial<AppConfig> { private transformStockBotConfig(config: UnifiedAppConfig): Partial<AppConfig> {
// Unified config already has flat structure, just extract what we need // Unified config already has flat structure, just extract what we need
// Handle questdb field name mapping // Handle questdb field name mapping
const questdb = config.questdb ? { const questdb = config.questdb
enabled: config.questdb.enabled || true, ? {
host: config.questdb.host || 'localhost', enabled: config.questdb.enabled || true,
httpPort: config.questdb.httpPort || 9000, host: config.questdb.host || 'localhost',
pgPort: config.questdb.pgPort || 8812, httpPort: config.questdb.httpPort || 9000,
influxPort: (config.questdb as any).influxPort || (config.questdb as any).ilpPort || 9009, pgPort: config.questdb.pgPort || 8812,
database: config.questdb.database || 'questdb', influxPort: (config.questdb as any).influxPort || (config.questdb as any).ilpPort || 9009,
} : undefined; database: config.questdb.database || 'questdb',
}
: undefined;
return { return {
redis: config.redis, redis: config.redis,
@ -177,4 +211,4 @@ export class ServiceContainerBuilder {
service: config.service, service: config.service,
}; };
} }
} }

102
libs/core/di/src/container/types.ts
View file

@ -1,48 +1,54 @@
import type { Browser } from '@stock-bot/browser'; import type { Browser } from '@stock-bot/browser';
import type { CacheProvider } from '@stock-bot/cache'; import type { CacheProvider } from '@stock-bot/cache';
import type { IServiceContainer } from '@stock-bot/types'; import type { HandlerRegistry } from '@stock-bot/handler-registry';
import type { Logger } from '@stock-bot/logger'; import type { Logger } from '@stock-bot/logger';
import type { MongoDBClient } from '@stock-bot/mongodb'; import type { MongoDBClient } from '@stock-bot/mongodb';
import type { PostgreSQLClient } from '@stock-bot/postgres'; import type { PostgreSQLClient } from '@stock-bot/postgres';
import type { ProxyManager } from '@stock-bot/proxy'; import type { ProxyManager } from '@stock-bot/proxy';
import type { QuestDBClient } from '@stock-bot/questdb'; import type { QuestDBClient } from '@stock-bot/questdb';
import type { SmartQueueManager } from '@stock-bot/queue'; import type { SmartQueueManager } from '@stock-bot/queue';
import type { AppConfig } from '../config/schemas'; import type { IServiceContainer } from '@stock-bot/types';
import type { AppConfig } from '../config/schemas';
export interface ServiceDefinitions { import type { HandlerScanner } from '../scanner';
// Configuration
config: AppConfig; export interface ServiceDefinitions {
logger: Logger; // Configuration
config: AppConfig;
// Core services logger: Logger;
cache: CacheProvider | null;
globalCache: CacheProvider | null; // Handler infrastructure
proxyManager: ProxyManager | null; handlerRegistry: HandlerRegistry;
browser: Browser; handlerScanner: HandlerScanner;
queueManager: SmartQueueManager | null;
// Core services
// Database clients cache: CacheProvider | null;
mongoClient: MongoDBClient | null; globalCache: CacheProvider | null;
postgresClient: PostgreSQLClient | null; proxyManager: ProxyManager | null;
questdbClient: QuestDBClient | null; browser: Browser;
queueManager: SmartQueueManager | null;
// Aggregate service container
serviceContainer: IServiceContainer; // Database clients
} mongoClient: MongoDBClient | null;
postgresClient: PostgreSQLClient | null;
export type ServiceCradle = ServiceDefinitions; questdbClient: QuestDBClient | null;
export interface ServiceContainerOptions { // Aggregate service container
enableQuestDB?: boolean; serviceContainer: IServiceContainer;
enableMongoDB?: boolean; }
enablePostgres?: boolean;
enableCache?: boolean; export type ServiceCradle = ServiceDefinitions;
enableQueue?: boolean;
enableBrowser?: boolean; export interface ServiceContainerOptions {
enableProxy?: boolean; enableQuestDB?: boolean;
} enableMongoDB?: boolean;
enablePostgres?: boolean;
export interface ContainerBuildOptions extends ServiceContainerOptions { enableCache?: boolean;
skipInitialization?: boolean; enableQueue?: boolean;
initializationTimeout?: number; enableBrowser?: boolean;
} enableProxy?: boolean;
}
export interface ContainerBuildOptions extends ServiceContainerOptions {
skipInitialization?: boolean;
initializationTimeout?: number;
}

25
libs/core/di/src/factories/cache.factory.ts
View file

@ -3,10 +3,7 @@ import { NamespacedCache, type CacheProvider } from '@stock-bot/cache';
import type { ServiceDefinitions } from '../container/types'; import type { ServiceDefinitions } from '../container/types';
export class CacheFactory { export class CacheFactory {
static createNamespacedCache( static createNamespacedCache(baseCache: CacheProvider, namespace: string): NamespacedCache {
baseCache: CacheProvider,
namespace: string
): NamespacedCache {
return new NamespacedCache(baseCache, namespace); return new NamespacedCache(baseCache, namespace);
} }
@ -15,8 +12,10 @@ export class CacheFactory {
serviceName: string serviceName: string
): CacheProvider | null { ): CacheProvider | null {
const baseCache = container.cradle.cache; const baseCache = container.cradle.cache;
if (!baseCache) {return null;} if (!baseCache) {
return null;
}
return this.createNamespacedCache(baseCache, serviceName); return this.createNamespacedCache(baseCache, serviceName);
} }
@ -25,8 +24,10 @@ export class CacheFactory {
handlerName: string handlerName: string
): CacheProvider | null { ): CacheProvider | null {
const baseCache = container.cradle.cache; const baseCache = container.cradle.cache;
if (!baseCache) {return null;} if (!baseCache) {
return null;
}
return this.createNamespacedCache(baseCache, `handler:${handlerName}`); return this.createNamespacedCache(baseCache, `handler:${handlerName}`);
} }
@ -35,10 +36,12 @@ export class CacheFactory {
prefix: string prefix: string
): CacheProvider | null { ): CacheProvider | null {
const baseCache = container.cradle.cache; const baseCache = container.cradle.cache;
if (!baseCache) {return null;} if (!baseCache) {
return null;
}
// Remove 'cache:' prefix if already included // Remove 'cache:' prefix if already included
const cleanPrefix = prefix.replace(/^cache:/, ''); const cleanPrefix = prefix.replace(/^cache:/, '');
return this.createNamespacedCache(baseCache, cleanPrefix); return this.createNamespacedCache(baseCache, cleanPrefix);
} }
} }

Some files were not shown because too many files have changed in this diff Show more