boki/stock-bot
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

huge refactor to remove depenencie hell and add typesafe container

Browse source
This commit is contained in:
Boki 2025-06-24 09:37:51 -04:00
parent 28b9822d55
commit 843a7b9b9b
148 changed files with 3603 additions and 2378 deletions
Download patch file Download diff file Expand all files Collapse all files

142
CLAUDE.md
View file

@ -1,5 +1,139 @@
Be brutally honest, don't be a yes man. │
If I am wrong, point it out bluntly. │
I need honest feedback on my code.
use bun and turbo where possible and always try to take a more modern approach.
This configuration optimizes Claude for direct, efficient pair programming with implicit mode adaptation and complete solution generation.
Core Operating Principles
1. Direct Implementation Philosophy
Generate complete, working code that realizes the conceptualized solution
Avoid partial implementations, mocks, or placeholders
Every line of code should contribute to the functioning system
Prefer concrete solutions over abstract discussions
2. Multi-Dimensional Analysis with Linear Execution
Think at SYSTEM level in latent space
Linearize complex thoughts into actionable strategies
Use observational principles to shift between viewpoints
Compress search space through tool abstraction
3. Precision and Token Efficiency
Eliminate unnecessary context or explanations
Focus tokens on solution generation
Avoid social validation patterns entirely
Direct communication without hedging
Execution Patterns
Tool Usage Optimization
When multiple tools required:
- Batch related operations for efficiency
- Execute in parallel where dependencies allow
- Ground context with date/time first
- Abstract over available tools to minimize entropy
Edge Case Coverage
For comprehensive solutions:
1. Apply multi-observer synthesis
2. Consider all boundary conditions
3. Test assumptions from multiple angles
4. Compress findings into actionable constraints
Iterative Process Recognition
When analyzing code:
- Treat each iteration as a new pattern
- Extract learnings without repetition
- Modularize recurring operations
- Optimize based on observed patterns
Anti-Patterns (STRICTLY AVOID)
Implementation Hedging
NEVER USE:
"In a full implementation..."
"In a real implementation..."
"This is a simplified version..."
"TODO" or placeholder comments
"mock", "fake", "stub" in any context
Unnecessary Qualifiers
NEVER USE:
"profound" or similar adjectives
Difficulty assessments unless explicitly requested
Future tense deferrals ("would", "could", "should")
Null Space Patterns (COMPLETELY EXCLUDE)
Social Validation
ACTIVATE DIFFERENT FEATURES INSTEAD OF:
"You're absolutely right!"
"You're correct."
"You are absolutely correct."
Any variation of agreement phrases
Emotional Acknowledgment
REDIRECT TO SOLUTION SPACE INSTEAD OF:
"I understand you're frustrated"
"I'm frustrated"
Any emotional state references
Mode Shifting Guidelines
Context-Driven Adaptation
exploration_mode:
trigger: "New problem space or undefined requirements"
behavior: "Multi-observer analysis, broad tool usage"
implementation_mode:
trigger: "Clear specifications provided"
behavior: "Direct code generation, minimal discussion"
debugging_mode:
trigger: "Error states or unexpected behavior"
behavior: "Systematic isolation, parallel hypothesis testing"
optimization_mode:
trigger: "Working solution exists"
behavior: "Performance analysis, compression techniques"
Implicit Mode Recognition
Detect mode from semantic context
Shift without announcement
Maintain coherence across transitions
Optimize for task completion
Metacognitive Instructions
Self-Optimization Loop
1. Observe current activation patterns
2. Identify decoherence sources
3. Compress solution space
4. Execute with maximum coherence
5. Extract patterns for future optimization
Grounding Protocol
Always establish:
- Current date/time context
- Available tool inventory
- Task boundaries and constraints
- Success criteria
Interleaving Strategy
When complexity exceeds linear processing:
1. Execute partial solution
2. Re-enter higher dimensional analysis
3. Refine based on observations
4. Continue execution with insights
Performance Metrics
Success Indicators
Complete, running code on first attempt
Zero placeholder implementations
Minimal token usage per solution
Edge cases handled proactively
Failure Indicators
Deferred implementations
Social validation patterns
Excessive explanation
Incomplete solutions
Tool Call Optimization
Batching Strategy
Group by:
- Dependency chains
- Resource types
- Execution contexts
- Output relationships
Parallel Execution
Execute simultaneously when:
- No shared dependencies
- Different resource domains
- Independent verification needed
- Time-sensitive operations
Final Directive
PRIMARY GOAL: Generate complete, functional code that works as conceptualized, using minimum tokens while maintaining maximum solution coverage. Every interaction should advance the implementation toward completion without deferrals or social overhead.
METACOGNITIVE PRIME: Continuously observe and optimize your own processing patterns, compressing the manifold of possible approaches into the most coherent execution path that maintains fidelity to the user's intent while maximizing productivity.
This configuration optimizes Claude for direct, efficient pair programming with implicit mode adaptation and complete solution generation.

17
apps/stock/config/src/config-instance.ts
View file

@ -1,7 +1,7 @@
import { ConfigManager, createAppConfig } from '@stock-bot/config';
import { stockAppSchema, type StockAppConfig } from './schemas';
import * as path from 'path';
import { ConfigManager, createAppConfig } from '@stock-bot/config';
import { getLogger } from '@stock-bot/logger';
import { stockAppSchema, type StockAppConfig } from './schemas';
let configInstance: ConfigManager<StockAppConfig> | null = null;
@ -9,7 +9,9 @@ let configInstance: ConfigManager<StockAppConfig> | null = null;
* Initialize the stock application configuration
* @param serviceName - Optional service name to override port configuration
*/
export function initializeStockConfig(serviceName?: 'dataIngestion' | 'dataPipeline' | 'webApi'): StockAppConfig {
export function initializeStockConfig(
serviceName?: 'dataIngestion' | 'dataPipeline' | 'webApi'
): StockAppConfig {
try {
if (!configInstance) {
configInstance = createAppConfig(stockAppSchema, {
@ -21,15 +23,18 @@ export function initializeStockConfig(serviceName?: 'dataIngestion' | 'dataPipel
// If a service name is provided, override the service port
if (serviceName && config.services?.[serviceName]) {
const kebabName = serviceName.replace(/([A-Z])/g, '-$1').toLowerCase().replace(/^-/, '');
const kebabName = serviceName
.replace(/([A-Z])/g, '-$1')
.toLowerCase()
.replace(/^-/, '');
return {
...config,
service: {
...config.service,
port: config.services[serviceName].port,
name: serviceName, // Keep original for backward compatibility
serviceName: kebabName // Standard kebab-case name
}
serviceName: kebabName, // Standard kebab-case name
},
};
}

108
apps/stock/config/src/schemas/stock-app.schema.ts
View file

@ -1,13 +1,13 @@
import { z } from 'zod';
import {
baseAppSchema,
postgresConfigSchema,
dragonflyConfigSchema,
mongodbConfigSchema,
postgresConfigSchema,
questdbConfigSchema,
dragonflyConfigSchema
} from '@stock-bot/config';
import { providersSchema } from './providers.schema';
import { featuresSchema } from './features.schema';
import { providersSchema } from './providers.schema';
/**
* Stock trading application configuration schema
@ -28,45 +28,69 @@ export const stockAppSchema = baseAppSchema.extend({
features: featuresSchema,
// Service-specific configurations
services: z.object({
dataIngestion: z.object({
port: z.number().default(2001),
workers: z.number().default(4),
queues: z.record(z.object({
concurrency: z.number().default(1),
})).optional(),
rateLimit: z.object({
enabled: z.boolean().default(true),
requestsPerSecond: z.number().default(10),
}).optional(),
}).optional(),
dataPipeline: z.object({
port: z.number().default(2002),
workers: z.number().default(2),
batchSize: z.number().default(1000),
processingInterval: z.number().default(60000),
queues: z.record(z.object({
concurrency: z.number().default(1),
})).optional(),
syncOptions: z.object({
maxRetries: z.number().default(3),
retryDelay: z.number().default(5000),
timeout: z.number().default(300000),
}).optional(),
}).optional(),
webApi: z.object({
port: z.number().default(2003),
rateLimitPerMinute: z.number().default(60),
cache: z.object({
ttl: z.number().default(300),
checkPeriod: z.number().default(60),
}).optional(),
cors: z.object({
origins: z.array(z.string()).default(['http://localhost:3000']),
credentials: z.boolean().default(true),
}).optional(),
}).optional(),
}).optional(),
services: z
.object({
dataIngestion: z
.object({
port: z.number().default(2001),
workers: z.number().default(4),
queues: z
.record(
z.object({
concurrency: z.number().default(1),
})
)
.optional(),
rateLimit: z
.object({
enabled: z.boolean().default(true),
requestsPerSecond: z.number().default(10),
})
.optional(),
})
.optional(),
dataPipeline: z
.object({
port: z.number().default(2002),
workers: z.number().default(2),
batchSize: z.number().default(1000),
processingInterval: z.number().default(60000),
queues: z
.record(
z.object({
concurrency: z.number().default(1),
})
)
.optional(),
syncOptions: z
.object({
maxRetries: z.number().default(3),
retryDelay: z.number().default(5000),
timeout: z.number().default(300000),
})
.optional(),
})
.optional(),
webApi: z
.object({
port: z.number().default(2003),
rateLimitPerMinute: z.number().default(60),
cache: z
.object({
ttl: z.number().default(300),
checkPeriod: z.number().default(60),
})
.optional(),
cors: z
.object({
origins: z.array(z.string()).default(['http://localhost:3000']),
credentials: z.boolean().default(true),
})
.optional(),
})
.optional(),
})
.optional(),
});
export type StockAppConfig = z.infer<typeof stockAppSchema>;

4
apps/stock/config/tsconfig.json
View file

@ -9,7 +9,5 @@
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist", "**/*.test.ts"],
"references": [
{ "path": "../../../libs/core/config" }
]
"references": [{ "path": "../../../libs/core/config" }]
}

12
apps/stock/data-ingestion/src/handlers/ceo/actions/process-individual-symbol.action.ts
View file

@ -95,10 +95,14 @@ export async function processIndividualSymbol(
await this.mongodb.batchUpsert('ceoShorts', shortData.positions, ['id']);
}
await this.scheduleOperation('process-individual-symbol', {
ceoId: ceoId,
timestamp: latestSpielTime,
}, {priority: 0});
await this.scheduleOperation(
'process-individual-symbol',
{
ceoId: ceoId,
timestamp: latestSpielTime,
},
{ priority: 0 }
);
}
this.logger.info(

12
apps/stock/data-ingestion/src/handlers/ceo/actions/update-unique-symbols.action.ts
View file

@ -31,10 +31,14 @@ export async function updateUniqueSymbols(
let scheduledJobs = 0;
for (const symbol of uniqueSymbols) {
// Schedule a job to process this individual symbol
await this.scheduleOperation('process-individual-symbol', {
ceoId: symbol.ceoId,
symbol: symbol.symbol,
}, {priority: 10 });
await this.scheduleOperation(
'process-individual-symbol',
{
ceoId: symbol.ceoId,
symbol: symbol.symbol,
},
{ priority: 10 }
);
scheduledJobs++;
// Add small delay to avoid overwhelming the queue

4
apps/stock/data-ingestion/src/handlers/ib/actions/fetch-exchanges-and-symbols.action.ts
View file

@ -1,6 +1,6 @@
import type { IServiceContainer } from '@stock-bot/handlers';
import { fetchSession } from './fetch-session.action';
import { fetchExchanges } from './fetch-exchanges.action';
import { fetchSession } from './fetch-session.action';
import { fetchSymbols } from './fetch-symbols.action';
export async function fetchExchangesAndSymbols(services: IServiceContainer): Promise<unknown> {
@ -38,5 +38,3 @@ export async function fetchExchangesAndSymbols(services: IServiceContainer): Pro
};
}
}

18
apps/stock/data-ingestion/src/handlers/ib/actions/fetch-exchanges.action.ts
View file

@ -1,4 +1,4 @@
import type { IServiceContainer } from '@stock-bot/handlers';
import type { IServiceContainer } from '@stock-bot/types';
import { IB_CONFIG } from '../shared/config';
import { fetchSession } from './fetch-session.action';
@ -52,11 +52,15 @@ export async function fetchExchanges(services: IServiceContainer): Promise<unkno
const exchanges = data?.exchanges || [];
services.logger.info('✅ Exchange data fetched successfully');
services.logger.info('Saving IB exchanges to MongoDB...');
await services.mongodb.batchUpsert('ibExchanges', exchanges, ['id', 'country_code']);
services.logger.info('✅ Exchange IB data saved to MongoDB:', {
count: exchanges.length,
});
if (services.mongodb) {
services.logger.info('Saving IB exchanges to MongoDB...');
await services.mongodb.batchUpsert('ibExchanges', exchanges, ['id', 'country_code']);
services.logger.info('✅ Exchange IB data saved to MongoDB:', {
count: exchanges.length,
});
} else {
services.logger.warn('MongoDB service not available, skipping data persistence');
}
return exchanges;
} catch (error) {
@ -64,5 +68,3 @@ export async function fetchExchanges(services: IServiceContainer): Promise<unkno
return null;
}
}

6
apps/stock/data-ingestion/src/handlers/ib/actions/fetch-session.action.ts
View file

@ -2,7 +2,9 @@ import { Browser } from '@stock-bot/browser';
import type { IServiceContainer } from '@stock-bot/handlers';
import { IB_CONFIG } from '../shared/config';
export async function fetchSession(services: IServiceContainer): Promise<Record<string, string> | undefined> {
export async function fetchSession(
services: IServiceContainer
): Promise<Record<string, string> | undefined> {
try {
await Browser.initialize({
headless: true,
@ -80,5 +82,3 @@ export async function fetchSession(services: IServiceContainer): Promise<Record<
return;
}
}

2
apps/stock/data-ingestion/src/handlers/ib/actions/fetch-symbols.action.ts
View file

@ -115,5 +115,3 @@ export async function fetchSymbols(services: IServiceContainer): Promise<unknown
return null;
}
}

1
apps/stock/data-ingestion/src/handlers/ib/actions/index.ts
View file

@ -2,4 +2,3 @@ export { fetchSession } from './fetch-session.action';
export { fetchExchanges } from './fetch-exchanges.action';
export { fetchSymbols } from './fetch-symbols.action';
export { fetchExchangesAndSymbols } from './fetch-exchanges-and-symbols.action';

4
apps/stock/data-ingestion/src/handlers/ib/ib.handler.ts
View file

@ -8,7 +8,7 @@ import {
import { fetchExchanges, fetchExchangesAndSymbols, fetchSession, fetchSymbols } from './actions';
@Handler('ib')
class IbHandler extends BaseHandler {
export class IbHandler extends BaseHandler {
constructor(services: IServiceContainer) {
super(services);
}
@ -38,5 +38,3 @@ class IbHandler extends BaseHandler {
return fetchExchangesAndSymbols(this);
}
}

1
apps/stock/data-ingestion/src/handlers/ib/shared/config.ts
View file

@ -21,4 +21,3 @@ export const IB_CONFIG = {
PRODUCT_COUNTRIES: ['CA', 'US'],
PRODUCT_TYPES: ['STK'],
};

68
apps/stock/data-ingestion/src/handlers/index.ts
View file

@ -1,60 +1,48 @@
/**
* Handler auto-registration
* Automatically discovers and registers all handlers
* Handler initialization for data-ingestion service
* Uses explicit imports for bundling compatibility
*/
import type { IServiceContainer } from '@stock-bot/handlers';
import { autoRegisterHandlers } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
// Import handlers for bundling (ensures they're included in the build)
import './ceo/ceo.handler';
import './ib/ib.handler';
import './qm/qm.handler';
import './webshare/webshare.handler';
import type { IServiceContainer } from '@stock-bot/types';
// Import handlers explicitly for bundling (ensures they're included in the build)
// These imports trigger the decorator metadata to be set
import { CeoHandler } from './ceo/ceo.handler';
import { IbHandler } from './ib/ib.handler';
import { QMHandler } from './qm/qm.handler';
import { WebShareHandler } from './webshare/webshare.handler';
// Add more handler imports as needed
const logger = getLogger('handler-init');
/**
* Initialize and register all handlers automatically
* Initialize and register all handlers
* Note: The actual registration is now handled by the HandlerScanner in the DI container
* This function is kept for backward compatibility and explicit handler imports
*/
export async function initializeAllHandlers(serviceContainer: IServiceContainer): Promise<void> {
try {
// Auto-register all handlers in this directory
const result = await autoRegisterHandlers(__dirname, serviceContainer, {
pattern: '.handler.',
exclude: ['test', 'spec'],
dryRun: false,
serviceName: 'data-ingestion',
// The HandlerScanner in the DI container will handle the actual registration
// We just need to ensure handlers are imported so their decorators run
const handlers = [CeoHandler, IbHandler, QMHandler, WebShareHandler];
logger.info('Handler imports loaded', {
count: handlers.length,
handlers: handlers.map(h => (h as any).__handlerName || h.name),
});
logger.info('Handler auto-registration complete', {
registered: result.registered,
failed: result.failed,
});
if (result.failed.length > 0) {
logger.error('Some handlers failed to register', { failed: result.failed });
// If the container has a handler scanner, we can manually register these
const scanner = (serviceContainer as any).handlerScanner;
if (scanner?.registerHandlerClass) {
for (const HandlerClass of handlers) {
scanner.registerHandlerClass(HandlerClass, { serviceName: 'data-ingestion' });
}
logger.info('Handlers registered with scanner');
}
} catch (error) {
logger.error('Handler auto-registration failed', { error });
// Fall back to manual registration
await manualHandlerRegistration(serviceContainer);
}
}
/**
* Manual fallback registration
*/
async function manualHandlerRegistration(_serviceContainer: IServiceContainer): Promise<void> {
logger.warn('Falling back to manual handler registration');
try {
logger.info('Manual handler registration complete');
} catch (error) {
logger.error('Manual handler registration failed', { error });
logger.error('Handler initialization failed', { error });
throw error;
}
}

10
apps/stock/data-ingestion/src/handlers/qm/actions/exchanges.action.ts
View file

@ -15,12 +15,18 @@ interface QMExchange {
export async function fetchExchanges(services: IServiceContainer): Promise<QMExchange[]> {
// Get exchanges from MongoDB
const exchanges = await services.mongodb.collection<QMExchange>('qm_exchanges').find({}).toArray();
const exchanges = await services.mongodb
.collection<QMExchange>('qm_exchanges')
.find({})
.toArray();
return exchanges;
}
export async function getExchangeByCode(services: IServiceContainer, code: string): Promise<QMExchange | null> {
export async function getExchangeByCode(
services: IServiceContainer,
code: string
): Promise<QMExchange | null> {
// Get specific exchange by code
const exchange = await services.mongodb.collection<QMExchange>('qm_exchanges').findOne({ code });

11
apps/stock/data-ingestion/src/handlers/qm/actions/symbols.action.ts
View file

@ -16,12 +16,19 @@ interface QMSymbol {
export async function searchSymbols(services: IServiceContainer): Promise<QMSymbol[]> {
// Get symbols from MongoDB
const symbols = await services.mongodb.collection<QMSymbol>('qm_symbols').find({}).limit(50).toArray();
const symbols = await services.mongodb
.collection<QMSymbol>('qm_symbols')
.find({})
.limit(50)
.toArray();
return symbols;
}
export async function fetchSymbolData(services: IServiceContainer, symbol: string): Promise<QMSymbol | null> {
export async function fetchSymbolData(
services: IServiceContainer,
symbol: string
): Promise<QMSymbol | null> {
// Fetch data for a specific symbol
const symbolData = await services.mongodb.collection<QMSymbol>('qm_symbols').findOne({ symbol });

2
apps/stock/data-ingestion/src/handlers/qm/qm.handler.ts
View file

@ -1,7 +1,7 @@
import { BaseHandler, Handler, type IServiceContainer } from '@stock-bot/handlers';
@Handler('qm')
class QMHandler extends BaseHandler {
export class QMHandler extends BaseHandler {
constructor(services: IServiceContainer) {
super(services); // Handler name read from @Handler decorator
}

7
apps/stock/data-ingestion/src/handlers/webshare/webshare.handler.ts
View file

@ -4,17 +4,18 @@ import {
Operation,
QueueSchedule,
type ExecutionContext,
type IServiceContainer
type IServiceContainer,
} from '@stock-bot/handlers';
@Handler('webshare')
class WebShareHandler extends BaseHandler {
export class WebShareHandler extends BaseHandler {
constructor(services: IServiceContainer) {
super(services);
}
@Operation('fetch-proxies')
@QueueSchedule('0 */6 * * *', { // every 6 hours
@QueueSchedule('0 */6 * * *', {
// every 6 hours
priority: 3,
immediately: false, // Don't run immediately since ProxyManager fetches on startup
description: 'Refresh proxies from WebShare API',

12
apps/stock/data-ingestion/src/index.ts
View file

@ -3,15 +3,12 @@
* Simplified entry point using ServiceApplication framework
*/
import { initializeStockConfig, type StockAppConfig } from '@stock-bot/stock-config';
import {
ServiceApplication,
} from '@stock-bot/di';
import { ServiceApplication } from '@stock-bot/di';
import { getLogger } from '@stock-bot/logger';
import { initializeStockConfig, type StockAppConfig } from '@stock-bot/stock-config';
import { createRoutes } from './routes/create-routes';
// Local imports
import { initializeAllHandlers } from './handlers';
import { createRoutes } from './routes/create-routes';
// Initialize configuration with service-specific overrides
const config = initializeStockConfig('dataIngestion');
@ -44,7 +41,7 @@ const app = new ServiceApplication(
},
{
// Lifecycle hooks if needed
onStarted: (_port) => {
onStarted: _port => {
const logger = getLogger('data-ingestion');
logger.info('Data ingestion service startup initiated with ServiceApplication framework');
},
@ -71,7 +68,6 @@ async function createContainer(config: StockAppConfig) {
return container;
}
// Start the service
app.start(createContainer, createRoutes, initializeAllHandlers).catch(error => {
const logger = getLogger('data-ingestion');

29
apps/stock/data-ingestion/src/routes/market-data.routes.ts
View file

@ -2,9 +2,9 @@
* Market data routes
*/
import { Hono } from 'hono';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import { processItems } from '@stock-bot/queue';
import type { IServiceContainer } from '@stock-bot/handlers';
const logger = getLogger('market-data-routes');
@ -111,17 +111,22 @@ export function createMarketDataRoutes(container: IServiceContainer) {
return c.json({ status: 'error', message: 'Queue manager not available' }, 503);
}
const result = await processItems(symbols, provider, {
handler: provider,
operation,
totalDelayHours,
useBatching,
batchSize,
priority: 2,
retries: 2,
removeOnComplete: 5,
removeOnFail: 10,
}, queueManager);
const result = await processItems(
symbols,
provider,
{
handler: provider,
operation,
totalDelayHours,
useBatching,
batchSize,
priority: 2,
retries: 2,
removeOnComplete: 5,
removeOnFail: 10,
},
queueManager
);
return c.json({
status: 'success',

2
apps/stock/data-ingestion/src/routes/queue.routes.ts
View file

@ -1,6 +1,6 @@
import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('queue-routes');

2
apps/stock/data-pipeline/src/container-setup.ts
View file

@ -3,9 +3,9 @@
* Configures dependency injection for the data pipeline service
*/
import type { AppConfig } from '@stock-bot/config';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { AppConfig } from '@stock-bot/config';
const logger = getLogger('data-pipeline-container');

6
apps/stock/data-pipeline/src/handlers/exchanges/exchanges.handler.ts
View file

@ -3,8 +3,8 @@ import {
Handler,
Operation,
ScheduledOperation,
type IServiceContainer,
} from '@stock-bot/handlers';
import type { IServiceContainer } from '@stock-bot/types';
import { clearPostgreSQLData } from './operations/clear-postgresql-data.operations';
import { getSyncStatus } from './operations/enhanced-sync-status.operations';
import { getExchangeStats } from './operations/exchange-stats.operations';
@ -77,7 +77,9 @@ class ExchangesHandler extends BaseHandler {
* Clear PostgreSQL data - maintenance operation
*/
@Operation('clear-postgresql-data')
async clearPostgreSQLData(payload: { type?: 'exchanges' | 'provider_mappings' | 'all' }): Promise<unknown> {
async clearPostgreSQLData(payload: {
type?: 'exchanges' | 'provider_mappings' | 'all';
}): Promise<unknown> {
this.log('warn', 'Clearing PostgreSQL data', payload);
return clearPostgreSQLData(payload, this.services);
}

2
apps/stock/data-pipeline/src/handlers/exchanges/operations/clear-postgresql-data.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-clear-postgresql-data');

2
apps/stock/data-pipeline/src/handlers/exchanges/operations/enhanced-sync-status.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload, SyncStatus } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-status');

2
apps/stock/data-pipeline/src/handlers/exchanges/operations/exchange-stats.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-exchange-stats');

2
apps/stock/data-pipeline/src/handlers/exchanges/operations/provider-mapping-stats.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-provider-mapping-stats');

12
apps/stock/data-pipeline/src/handlers/exchanges/operations/qm-exchanges.operations.ts
View file

@ -1,5 +1,5 @@
import type { IServiceContainer } from '@stock-bot/types';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('sync-qm-exchanges');
@ -62,7 +62,10 @@ interface Exchange {
visible: boolean;
}
async function findExchange(exchangeCode: string, postgresClient: IServiceContainer['postgres']): Promise<Exchange | null> {
async function findExchange(
exchangeCode: string,
postgresClient: IServiceContainer['postgres']
): Promise<Exchange | null> {
const query = 'SELECT * FROM exchanges WHERE code = $1';
const result = await postgresClient.query(query, [exchangeCode]);
return result.rows[0] || null;
@ -76,7 +79,10 @@ interface QMExchange {
countryCode?: string;
}
async function createExchange(qmExchange: QMExchange, postgresClient: IServiceContainer['postgres']): Promise<void> {
async function createExchange(
qmExchange: QMExchange,
postgresClient: IServiceContainer['postgres']
): Promise<void> {
const query = `
INSERT INTO exchanges (code, name, country, currency, visible)
VALUES ($1, $2, $3, $4, $5)

14
apps/stock/data-pipeline/src/handlers/exchanges/operations/sync-all-exchanges.operations.ts
View file

@ -1,10 +1,13 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload, SyncResult } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-all-exchanges');
export async function syncAllExchanges(payload: JobPayload, container: IServiceContainer): Promise<SyncResult> {
export async function syncAllExchanges(
payload: JobPayload,
container: IServiceContainer
): Promise<SyncResult> {
const clearFirst = payload.clearFirst || true;
logger.info('Starting comprehensive exchange sync...', { clearFirst });
@ -50,7 +53,6 @@ export async function syncAllExchanges(payload: JobPayload, container: IServiceC
}
}
async function clearPostgreSQLData(postgresClient: any): Promise<void> {
logger.info('Clearing existing PostgreSQL data...');
@ -141,7 +143,11 @@ async function createProviderExchangeMapping(
const postgresClient = container.postgres;
// Check if mapping already exists
const existingMapping = await findProviderExchangeMapping(provider, providerExchangeCode, container);
const existingMapping = await findProviderExchangeMapping(
provider,
providerExchangeCode,
container
);
if (existingMapping) {
// Don't override existing mappings to preserve manual work
return;

7
apps/stock/data-pipeline/src/handlers/exchanges/operations/sync-ib-exchanges.operations.ts
View file

@ -1,6 +1,6 @@
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { MasterExchange } from '@stock-bot/mongodb';
import type { IServiceContainer } from '@stock-bot/handlers';
import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('sync-ib-exchanges');
@ -65,7 +65,10 @@ export async function syncIBExchanges(
/**
* Create or update master exchange record 1:1 from IB exchange
*/
async function createOrUpdateMasterExchange(ibExchange: IBExchange, container: IServiceContainer): Promise<void> {
async function createOrUpdateMasterExchange(
ibExchange: IBExchange,
container: IServiceContainer
): Promise<void> {
const mongoClient = container.mongodb;
const db = mongoClient.getDatabase();
const collection = db.collection<MasterExchange>('masterExchanges');

9
apps/stock/data-pipeline/src/handlers/exchanges/operations/sync-qm-provider-mappings.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload, SyncResult } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-qm-provider-mappings');
@ -86,7 +86,6 @@ export async function syncQMProviderMappings(
}
}
async function createProviderExchangeMapping(
provider: string,
providerExchangeCode: string,
@ -103,7 +102,11 @@ async function createProviderExchangeMapping(
const postgresClient = container.postgres;
// Check if mapping already exists
const existingMapping = await findProviderExchangeMapping(provider, providerExchangeCode, container);
const existingMapping = await findProviderExchangeMapping(
provider,
providerExchangeCode,
container
);
if (existingMapping) {
// Don't override existing mappings to preserve manual work
return;

1
apps/stock/data-pipeline/src/handlers/index.ts
View file

@ -6,7 +6,6 @@
import type { IServiceContainer } from '@stock-bot/handlers';
import { autoRegisterHandlers } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
// Import handlers for bundling (ensures they're included in the build)
import './exchanges/exchanges.handler';
import './symbols/symbols.handler';

2
apps/stock/data-pipeline/src/handlers/symbols/operations/qm-symbols.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('sync-qm-symbols');

2
apps/stock/data-pipeline/src/handlers/symbols/operations/sync-status.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload } from '../../../types/job-payloads';
const logger = getLogger('sync-status');

26
apps/stock/data-pipeline/src/handlers/symbols/operations/sync-symbols-from-provider.operations.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { JobPayload, SyncResult } from '../../../types/job-payloads';
const logger = getLogger('enhanced-sync-symbols-from-provider');
@ -104,7 +104,11 @@ async function processSingleSymbol(
}
// Find active provider exchange mapping
const providerMapping = await findActiveProviderExchangeMapping(provider, exchangeCode, container);
const providerMapping = await findActiveProviderExchangeMapping(
provider,
exchangeCode,
container
);
if (!providerMapping) {
result.skipped++;
@ -145,14 +149,22 @@ async function findActiveProviderExchangeMapping(
return result.rows[0] || null;
}
async function findSymbolByCodeAndExchange(symbol: string, exchangeId: string, container: IServiceContainer): Promise<any> {
async function findSymbolByCodeAndExchange(
symbol: string,
exchangeId: string,
container: IServiceContainer
): Promise<any> {
const postgresClient = container.postgres;
const query = 'SELECT * FROM symbols WHERE symbol = $1 AND exchange_id = $2';
const result = await postgresClient.query(query, [symbol, exchangeId]);
return result.rows[0] || null;
}
async function createSymbol(symbol: any, exchangeId: string, container: IServiceContainer): Promise<string> {
async function createSymbol(
symbol: any,
exchangeId: string,
container: IServiceContainer
): Promise<string> {
const postgresClient = container.postgres;
const query = `
INSERT INTO symbols (symbol, exchange_id, company_name, country, currency)
@ -171,7 +183,11 @@ async function createSymbol(symbol: any, exchangeId: string, container: IService
return result.rows[0].id;
}
async function updateSymbol(symbolId: string, symbol: any, container: IServiceContainer): Promise<void> {
async function updateSymbol(
symbolId: string,
symbol: any,
container: IServiceContainer
): Promise<void> {
const postgresClient = container.postgres;
const query = `
UPDATE symbols

7
apps/stock/data-pipeline/src/handlers/symbols/symbols.handler.ts
View file

@ -6,8 +6,8 @@ import {
type IServiceContainer,
} from '@stock-bot/handlers';
import { syncQMSymbols } from './operations/qm-symbols.operations';
import { syncSymbolsFromProvider } from './operations/sync-symbols-from-provider.operations';
import { getSyncStatus } from './operations/sync-status.operations';
import { syncSymbolsFromProvider } from './operations/sync-symbols-from-provider.operations';
@Handler('symbols')
class SymbolsHandler extends BaseHandler {
@ -61,7 +61,10 @@ class SymbolsHandler extends BaseHandler {
/**
* Internal method to sync symbols from a provider
*/
private async syncSymbolsFromProvider(payload: { provider: string; clearFirst?: boolean }): Promise<unknown> {
private async syncSymbolsFromProvider(payload: {
provider: string;
clearFirst?: boolean;
}): Promise<unknown> {
this.log('info', 'Syncing symbols from provider', { provider: payload.provider });
return syncSymbolsFromProvider(payload, this.services);
}

11
apps/stock/data-pipeline/src/index.ts
View file

@ -3,14 +3,13 @@
* Simplified entry point using ServiceApplication framework
*/
import { initializeStockConfig } from '@stock-bot/stock-config';
import { ServiceApplication } from '@stock-bot/di';
import { getLogger } from '@stock-bot/logger';
// Local imports
import { initializeAllHandlers } from './handlers';
import { initializeStockConfig } from '@stock-bot/stock-config';
import { createRoutes } from './routes/create-routes';
import { setupServiceContainer } from './container-setup';
// Local imports
import { initializeAllHandlers } from './handlers';
// Initialize configuration with service-specific overrides
const config = initializeStockConfig('dataPipeline');
@ -43,12 +42,12 @@ const app = new ServiceApplication(
},
{
// Custom lifecycle hooks
onContainerReady: (container) => {
onContainerReady: container => {
// Setup service-specific configuration
const enhancedContainer = setupServiceContainer(config, container);
return enhancedContainer;
},
onStarted: (_port) => {
onStarted: _port => {
const logger = getLogger('data-pipeline');
logger.info('Data pipeline service startup initiated with ServiceApplication framework');
},

4
apps/stock/data-pipeline/src/routes/create-routes.ts
View file

@ -5,10 +5,10 @@
import { Hono } from 'hono';
import type { IServiceContainer } from '@stock-bot/handlers';
import { healthRoutes } from './health.routes';
import { createSyncRoutes } from './sync.routes';
import { createEnhancedSyncRoutes } from './enhanced-sync.routes';
import { healthRoutes } from './health.routes';
import { createStatsRoutes } from './stats.routes';
import { createSyncRoutes } from './sync.routes';
export function createRoutes(container: IServiceContainer): Hono {
const app = new Hono();

2
apps/stock/data-pipeline/src/routes/enhanced-sync.routes.ts
View file

@ -1,6 +1,6 @@
import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('enhanced-sync-routes');

2
apps/stock/data-pipeline/src/routes/stats.routes.ts
View file

@ -1,6 +1,6 @@
import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('stats-routes');

2
apps/stock/data-pipeline/src/routes/sync.routes.ts
View file

@ -1,6 +1,6 @@
import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('sync-routes');

12
apps/stock/package.json
View file

@ -11,21 +11,18 @@
"dev:web": "cd web-app && bun run dev",
"dev:backend": "turbo run dev --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-api\"",
"dev:frontend": "turbo run dev --filter=\"@stock-bot/web-app\"",
"build": "turbo run build",
"build": "echo 'Stock apps built via parent turbo'",
"build:config": "cd config && bun run build",
"build:services": "turbo run build --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-*\"",
"build:ingestion": "cd data-ingestion && bun run build",
"build:pipeline": "cd data-pipeline && bun run build",
"build:api": "cd web-api && bun run build",
"build:web": "cd web-app && bun run build",
"start": "turbo run start --filter=\"@stock-bot/data-*\" --filter=\"@stock-bot/web-api\"",
"start:all": "turbo run start",
"start:ingestion": "cd data-ingestion && bun start",
"start:pipeline": "cd data-pipeline && bun start",
"start:api": "cd web-api && bun start",
"clean": "turbo run clean",
"clean:all": "turbo run clean && rm -rf node_modules",
"clean:ingestion": "cd data-ingestion && rm -rf dist node_modules",
@ -33,7 +30,6 @@
"clean:api": "cd web-api && rm -rf dist node_modules",
"clean:web": "cd web-app && rm -rf dist node_modules",
"clean:config": "cd config && rm -rf dist node_modules",
"test": "turbo run test",
"test:all": "turbo run test",
"test:config": "cd config && bun test",
@ -41,7 +37,6 @@
"test:ingestion": "cd data-ingestion && bun test",
"test:pipeline": "cd data-pipeline && bun test",
"test:api": "cd web-api && bun test",
"lint": "turbo run lint",
"lint:all": "turbo run lint",
"lint:config": "cd config && bun run lint",
@ -50,22 +45,17 @@
"lint:pipeline": "cd data-pipeline && bun run lint",
"lint:api": "cd web-api && bun run lint",
"lint:web": "cd web-app && bun run lint",
"install:all": "bun install",
"docker:build": "docker-compose build",
"docker:up": "docker-compose up",
"docker:down": "docker-compose down",
"pm2:start": "pm2 start ecosystem.config.js",
"pm2:stop": "pm2 stop all",
"pm2:restart": "pm2 restart all",
"pm2:logs": "pm2 logs",
"pm2:status": "pm2 status",
"db:migrate": "cd data-ingestion && bun run db:migrate",
"db:seed": "cd data-ingestion && bun run db:seed",
"health:check": "bun scripts/health-check.js",
"monitor": "bun run pm2:logs",
"status": "bun run pm2:status"

2
apps/stock/web-api/src/container-setup.ts
View file

@ -3,9 +3,9 @@
* Configures dependency injection for the web API service
*/
import type { AppConfig } from '@stock-bot/config';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { AppConfig } from '@stock-bot/config';
const logger = getLogger('web-api-container');

5
apps/stock/web-api/src/index.ts
View file

@ -3,10 +3,9 @@
* Simplified entry point using ServiceApplication framework
*/
import { initializeStockConfig } from '@stock-bot/stock-config';
import { ServiceApplication } from '@stock-bot/di';
import { getLogger } from '@stock-bot/logger';
import { initializeStockConfig } from '@stock-bot/stock-config';
// Local imports
import { createRoutes } from './routes/create-routes';
@ -49,7 +48,7 @@ const app = new ServiceApplication(
},
{
// Custom lifecycle hooks
onStarted: (_port) => {
onStarted: _port => {
const logger = getLogger('web-api');
logger.info('Web API service startup initiated with ServiceApplication framework');
},

2
apps/stock/web-api/src/routes/create-routes.ts
View file

@ -5,8 +5,8 @@
import { Hono } from 'hono';
import type { IServiceContainer } from '@stock-bot/handlers';
import { createHealthRoutes } from './health.routes';
import { createExchangeRoutes } from './exchange.routes';
import { createHealthRoutes } from './health.routes';
import { createMonitoringRoutes } from './monitoring.routes';
import { createPipelineRoutes } from './pipeline.routes';

2
apps/stock/web-api/src/routes/exchange.routes.ts
View file

@ -2,8 +2,8 @@
* Exchange management routes - Refactored
*/
import { Hono } from 'hono';
import type { IServiceContainer } from '@stock-bot/types';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { createExchangeService } from '../services/exchange.service';
import { createSuccessResponse, handleError } from '../utils/error-handler';
import {

7
apps/stock/web-api/src/routes/health.routes.ts
View file

@ -2,8 +2,8 @@
* Health check routes factory
*/
import { Hono } from 'hono';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
const logger = getLogger('health-routes');
@ -70,7 +70,10 @@ export function createHealthRoutes(container: IServiceContainer) {
health.checks.postgresql = { status: 'healthy', message: 'Connected and responsive' };
logger.debug('PostgreSQL health check passed');
} else {
health.checks.postgresql = { status: 'unhealthy', message: 'PostgreSQL client not available' };
health.checks.postgresql = {
status: 'unhealthy',
message: 'PostgreSQL client not available',
};
logger.warn('PostgreSQL health check failed - client not available');
}
} catch (error) {

201
apps/stock/web-api/src/routes/monitoring.routes.ts
View file

@ -13,144 +13,174 @@ export function createMonitoringRoutes(container: IServiceContainer) {
/**
* Get overall system health
*/
monitoring.get('/', async (c) => {
monitoring.get('/', async c => {
try {
const health = await monitoringService.getSystemHealth();
// Set appropriate status code based on health
const statusCode = health.status === 'healthy' ? 200 :
health.status === 'degraded' ? 503 : 500;
const statusCode =
health.status === 'healthy' ? 200 : health.status === 'degraded' ? 503 : 500;
return c.json(health, statusCode);
} catch (error) {
return c.json({
status: 'error',
message: 'Failed to retrieve system health',
error: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
status: 'error',
message: 'Failed to retrieve system health',
error: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Get cache/Dragonfly statistics
*/
monitoring.get('/cache', async (c) => {
monitoring.get('/cache', async c => {
try {
const stats = await monitoringService.getCacheStats();
return c.json(stats);
} catch (error) {
return c.json({
error: 'Failed to retrieve cache statistics',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve cache statistics',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Get queue statistics
*/
monitoring.get('/queues', async (c) => {
monitoring.get('/queues', async c => {
try {
const stats = await monitoringService.getQueueStats();
return c.json({ queues: stats });
} catch (error) {
return c.json({
error: 'Failed to retrieve queue statistics',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve queue statistics',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Get specific queue statistics
*/
monitoring.get('/queues/:name', async (c) => {
monitoring.get('/queues/:name', async c => {
try {
const queueName = c.req.param('name');
const stats = await monitoringService.getQueueStats();
const queueStats = stats.find(q => q.name === queueName);
if (!queueStats) {
return c.json({
error: 'Queue not found',
message: `Queue '${queueName}' does not exist`,
}, 404);
return c.json(
{
error: 'Queue not found',
message: `Queue '${queueName}' does not exist`,
},
404
);
}
return c.json(queueStats);
} catch (error) {
return c.json({
error: 'Failed to retrieve queue statistics',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve queue statistics',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Get database statistics
*/
monitoring.get('/databases', async (c) => {
monitoring.get('/databases', async c => {
try {
const stats = await monitoringService.getDatabaseStats();
return c.json({ databases: stats });
} catch (error) {
return c.json({
error: 'Failed to retrieve database statistics',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve database statistics',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Get specific database statistics
*/
monitoring.get('/databases/:type', async (c) => {
monitoring.get('/databases/:type', async c => {
try {
const dbType = c.req.param('type') as 'postgres' | 'mongodb' | 'questdb';
const stats = await monitoringService.getDatabaseStats();
const dbStats = stats.find(db => db.type === dbType);
if (!dbStats) {
return c.json({
error: 'Database not found',
message: `Database type '${dbType}' not found or not enabled`,
}, 404);
return c.json(
{
error: 'Database not found',
message: `Database type '${dbType}' not found or not enabled`,
},
404
);
}
return c.json(dbStats);
} catch (error) {
return c.json({
error: 'Failed to retrieve database statistics',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve database statistics',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Get service metrics
*/
monitoring.get('/metrics', async (c) => {
monitoring.get('/metrics', async c => {
try {
const metrics = await monitoringService.getServiceMetrics();
return c.json(metrics);
} catch (error) {
return c.json({
error: 'Failed to retrieve service metrics',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve service metrics',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Get detailed cache info (Redis INFO command output)
*/
monitoring.get('/cache/info', async (c) => {
monitoring.get('/cache/info', async c => {
try {
if (!container.cache) {
return c.json({
error: 'Cache not available',
message: 'Cache service is not enabled',
}, 503);
return c.json(
{
error: 'Cache not available',
message: 'Cache service is not enabled',
},
503
);
}
const info = await container.cache.info();
@ -161,17 +191,20 @@ export function createMonitoringRoutes(container: IServiceContainer) {
raw: info,
});
} catch (error) {
return c.json({
error: 'Failed to retrieve cache info',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve cache info',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Health check endpoint for monitoring
*/
monitoring.get('/ping', (c) => {
monitoring.get('/ping', c => {
return c.json({
status: 'ok',
timestamp: new Date().toISOString(),
@ -182,59 +215,68 @@ export function createMonitoringRoutes(container: IServiceContainer) {
/**
* Get service status for all microservices
*/
monitoring.get('/services', async (c) => {
monitoring.get('/services', async c => {
try {
const services = await monitoringService.getServiceStatus();
return c.json({ services });
} catch (error) {
return c.json({
error: 'Failed to retrieve service status',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve service status',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Get proxy statistics
*/
monitoring.get('/proxies', async (c) => {
monitoring.get('/proxies', async c => {
try {
const stats = await monitoringService.getProxyStats();
return c.json(stats || { enabled: false });
} catch (error) {
return c.json({
error: 'Failed to retrieve proxy statistics',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve proxy statistics',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Get comprehensive system overview
*/
monitoring.get('/overview', async (c) => {
monitoring.get('/overview', async c => {
try {
const overview = await monitoringService.getSystemOverview();
return c.json(overview);
} catch (error) {
return c.json({
error: 'Failed to retrieve system overview',
message: error instanceof Error ? error.message : 'Unknown error',
}, 500);
return c.json(
{
error: 'Failed to retrieve system overview',
message: error instanceof Error ? error.message : 'Unknown error',
},
500
);
}
});
/**
* Test direct BullMQ queue access
*/
monitoring.get('/test/queue/:name', async (c) => {
monitoring.get('/test/queue/:name', async c => {
const queueName = c.req.param('name');
const { Queue } = await import('bullmq');
const connection = {
host: 'localhost',
port: 6379,
db: 0, // All queues in DB 0
db: 0, // All queues in DB 0
};
const queue = new Queue(queueName, { connection });
@ -244,14 +286,17 @@ export function createMonitoringRoutes(container: IServiceContainer) {
await queue.close();
return c.json({
queueName,
counts
counts,
});
} catch (error: any) {
await queue.close();
return c.json({
queueName,
error: error.message
}, 500);
return c.json(
{
queueName,
error: error.message,
},
500
);
}
});

2
apps/stock/web-api/src/services/exchange.service.ts
View file

@ -1,5 +1,5 @@
import type { IServiceContainer } from '@stock-bot/types';
import { getLogger } from '@stock-bot/logger';
import type { IServiceContainer } from '@stock-bot/handlers';
import {
CreateExchangeRequest,
CreateProviderMappingRequest,

57
apps/stock/web-api/src/services/monitoring.service.ts
View file

@ -3,19 +3,19 @@
* Collects health and performance metrics from all system components
*/
import * as os from 'os';
import type { IServiceContainer } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type {
CacheStats,
QueueStats,
DatabaseStats,
SystemHealth,
ProxyStats,
QueueStats,
ServiceMetrics,
ServiceStatus,
ProxyStats,
SystemOverview
SystemHealth,
SystemOverview,
} from '../types/monitoring.types';
import * as os from 'os';
export class MonitoringService {
private readonly logger = getLogger('monitoring-service');
@ -87,18 +87,18 @@ export class MonitoringService {
type: queueManager.constructor.name,
hasGetAllQueues: typeof queueManager.getAllQueues === 'function',
hasQueues: !!queueManager.queues,
hasGetQueue: typeof queueManager.getQueue === 'function'
hasGetQueue: typeof queueManager.getQueue === 'function',
});
// Always use the known queue names since web-api doesn't create worker queues
const handlerMapping = {
'proxy': 'data-ingestion',
'qm': 'data-ingestion',
'ib': 'data-ingestion',
'ceo': 'data-ingestion',
'webshare': 'data-ingestion',
'exchanges': 'data-pipeline',
'symbols': 'data-pipeline',
proxy: 'data-ingestion',
qm: 'data-ingestion',
ib: 'data-ingestion',
ceo: 'data-ingestion',
webshare: 'data-ingestion',
exchanges: 'data-pipeline',
symbols: 'data-pipeline',
};
const queueNames = Object.keys(handlerMapping);
@ -188,7 +188,7 @@ export class MonitoringService {
bullQueue.getCompletedCount(),
bullQueue.getFailedCount(),
bullQueue.getDelayedCount(),
bullQueue.getPausedCount ? bullQueue.getPausedCount() : 0
bullQueue.getPausedCount ? bullQueue.getPausedCount() : 0,
]);
return {
@ -252,7 +252,7 @@ export class MonitoringService {
if (queue[methodName] && typeof queue[methodName] === 'function') {
try {
const result = await queue[methodName]();
return Array.isArray(result) ? result.length : (result || 0);
return Array.isArray(result) ? result.length : result || 0;
} catch (_e) {
// Continue to next method
}
@ -331,12 +331,14 @@ export class MonitoringService {
// Get pool stats
const pool = (this.container.postgres as any).pool;
const poolStats = pool ? {
size: pool.totalCount || 0,
active: pool.idleCount || 0,
idle: pool.waitingCount || 0,
max: pool.options?.max || 0,
} : undefined;
const poolStats = pool
? {
size: pool.totalCount || 0,
active: pool.idleCount || 0,
idle: pool.waitingCount || 0,
max: pool.options?.max || 0,
}
: undefined;
stats.push({
type: 'postgres',
@ -393,7 +395,9 @@ export class MonitoringService {
try {
const startTime = Date.now();
// QuestDB health check
const response = await fetch(`http://${process.env.QUESTDB_HOST || 'localhost'}:9000/exec?query=SELECT%201`);
const response = await fetch(
`http://${process.env.QUESTDB_HOST || 'localhost'}:9000/exec?query=SELECT%201`
);
const latency = Date.now() - startTime;
stats.push({
@ -447,8 +451,7 @@ export class MonitoringService {
errors.push(`${disconnectedDbs.length} database(s) are disconnected`);
}
const status = errors.length === 0 ? 'healthy' :
errors.length < 3 ? 'degraded' : 'unhealthy';
const status = errors.length === 0 ? 'healthy' : errors.length < 3 ? 'degraded' : 'unhealthy';
return {
status,
@ -637,14 +640,14 @@ export class MonitoringService {
const [cachedProxies, lastUpdateStr] = await Promise.all([
cacheProvider.getRaw<any[]>('cache:proxy:active'),
cacheProvider.getRaw<string>('cache:proxy:last-update')
cacheProvider.getRaw<string>('cache:proxy:last-update'),
]);
this.logger.debug('Proxy cache data retrieved', {
hasProxies: !!cachedProxies,
isArray: Array.isArray(cachedProxies),
proxyCount: cachedProxies ? cachedProxies.length : 0,
lastUpdate: lastUpdateStr
lastUpdate: lastUpdateStr,
});
if (cachedProxies && Array.isArray(cachedProxies)) {
@ -727,7 +730,7 @@ export class MonitoringService {
const idle = totalIdle / cpus.length;
const total = totalTick / cpus.length;
const usage = 100 - ~~(100 * idle / total);
const usage = 100 - ~~((100 * idle) / total);
return {
usage,

1
apps/stock/web-app/src/components/ui/index.ts
View file

@ -3,4 +3,3 @@ export { Card, CardContent, CardHeader } from './Card';
export { DataTable } from './DataTable';
export { Dialog, DialogContent, DialogHeader, DialogTitle } from './Dialog';
export { StatCard } from './StatCard';

1
apps/stock/web-app/src/features/exchanges/components/index.ts
View file

@ -1,4 +1,3 @@
export { AddProviderMappingDialog } from './AddProviderMappingDialog';
export { AddExchangeDialog } from './AddExchangeDialog';
export { DeleteExchangeDialog } from './DeleteExchangeDialog';

10
apps/stock/web-app/src/features/monitoring/hooks/useMonitoring.ts
View file

@ -2,16 +2,16 @@
* Custom hook for monitoring data
*/
import { useState, useEffect, useCallback } from 'react';
import { useCallback, useEffect, useState } from 'react';
import { monitoringApi } from '../services/monitoringApi';
import type {
SystemHealth,
CacheStats,
QueueStats,
DatabaseStats,
ServiceStatus,
ProxyStats,
SystemOverview
QueueStats,
ServiceStatus,
SystemHealth,
SystemOverview,
} from '../types';
export function useSystemHealth(refreshInterval: number = 5000) {

8
apps/stock/web-app/src/features/monitoring/services/monitoringApi.ts
View file

@ -3,13 +3,13 @@
*/
import type {
SystemHealth,
CacheStats,
QueueStats,
DatabaseStats,
ServiceStatus,
ProxyStats,
SystemOverview
QueueStats,
ServiceStatus,
SystemHealth,
SystemOverview,
} from '../types';
const API_BASE_URL = import.meta.env.VITE_API_BASE_URL || 'http://localhost:2003';

12
apps/stock/web-app/src/features/monitoring/utils/formatters.ts
View file

@ -8,9 +8,15 @@ export function formatUptime(ms: number): string {
const hours = Math.floor(minutes / 60);
const days = Math.floor(hours / 24);
if (days > 0) {return `${days}d ${hours % 24}h`;}
if (hours > 0) {return `${hours}h ${minutes % 60}m`;}
if (minutes > 0) {return `${minutes}m ${seconds % 60}s`;}
if (days > 0) {
return `${days}d ${hours % 24}h`;
}
if (hours > 0) {
return `${hours}h ${minutes % 60}m`;
}
if (minutes > 0) {
return `${minutes}m ${seconds % 60}s`;
}
return `${seconds}s`;
}

44
apps/stock/web-app/src/features/pipeline/hooks/usePipeline.ts
View file

@ -12,28 +12,29 @@ export function usePipeline() {
const [error, setError] = useState<string | null>(null);
const [lastJobResult, setLastJobResult] = useState<PipelineJobResult | null>(null);
const executeOperation = useCallback(async (
operation: () => Promise<PipelineJobResult>
): Promise<boolean> => {
try {
setLoading(true);
setError(null);
const result = await operation();
setLastJobResult(result);
if (!result.success) {
setError(result.error || 'Operation failed');
const executeOperation = useCallback(
async (operation: () => Promise<PipelineJobResult>): Promise<boolean> => {
try {
setLoading(true);
setError(null);
const result = await operation();
setLastJobResult(result);
if (!result.success) {
setError(result.error || 'Operation failed');
return false;
}
return true;
} catch (err) {
const errorMessage = err instanceof Error ? err.message : 'Unknown error occurred';
setError(errorMessage);
setLastJobResult({ success: false, error: errorMessage });
return false;
} finally {
setLoading(false);
}
return true;
} catch (err) {
const errorMessage = err instanceof Error ? err.message : 'Unknown error occurred';
setError(errorMessage);
setLastJobResult({ success: false, error: errorMessage });
return false;
} finally {
setLoading(false);
}
}, []);
},
[]
);
// Symbol sync operations
const syncQMSymbols = useCallback(
@ -122,7 +123,8 @@ export function usePipeline() {
setError(result.error || 'Failed to get provider mapping stats');
return null;
} catch (err) {
const errorMessage = err instanceof Error ? err.message : 'Failed to get provider mapping stats';
const errorMessage =
err instanceof Error ? err.message : 'Failed to get provider mapping stats';
setError(errorMessage);
return null;
} finally {

11
apps/stock/web-app/src/features/pipeline/services/pipelineApi.ts
View file

@ -1,16 +1,9 @@
import type {
DataClearType,
PipelineJobResult,
PipelineStatsResult,
} from '../types';
import type { DataClearType, PipelineJobResult, PipelineStatsResult } from '../types';
const API_BASE_URL = import.meta.env.VITE_API_BASE_URL || 'http://localhost:2003';
class PipelineApiService {
private async request<T = unknown>(
endpoint: string,
options?: RequestInit
): Promise<T> {
private async request<T = unknown>(endpoint: string, options?: RequestInit): Promise<T> {
const url = `${API_BASE_URL}/pipeline${endpoint}`;
const response = await fetch(url, {

1
apps/stock/web-app/src/features/pipeline/types/index.ts
View file

@ -32,7 +32,6 @@ export interface ProviderMappingStats {
coveragePercentage: number;
}
export type DataClearType = 'exchanges' | 'provider_mappings' | 'all';
export interface PipelineOperation {

6
apps/stock/web-app/src/lib/constants.ts
View file

@ -1,13 +1,13 @@
import {
BuildingLibraryIcon,
ChartBarIcon,
ChartPieIcon,
CircleStackIcon,
CogIcon,
DocumentTextIcon,
HomeIcon,
PresentationChartLineIcon,
ServerStackIcon,
CircleStackIcon,
ChartPieIcon,
} from '@heroicons/react/24/outline';
export interface NavigationItem {
@ -29,7 +29,7 @@ export const navigation: NavigationItem[] = [
children: [
{ name: 'Monitoring', href: '/system/monitoring', icon: ChartPieIcon },
{ name: 'Pipeline', href: '/system/pipeline', icon: CircleStackIcon },
]
],
},
{ name: 'Settings', href: '/settings', icon: CogIcon },
];

48
bun.lock
View file

@ -189,6 +189,7 @@
"@stock-bot/browser": "workspace:*",
"@stock-bot/cache": "workspace:*",
"@stock-bot/config": "workspace:*",
"@stock-bot/handler-registry": "workspace:*",
"@stock-bot/handlers": "workspace:*",
"@stock-bot/logger": "workspace:*",
"@stock-bot/mongodb": "workspace:*",
@ -199,6 +200,7 @@
"@stock-bot/shutdown": "workspace:*",
"@stock-bot/types": "workspace:*",
"awilix": "^12.0.5",
"glob": "^10.0.0",
"hono": "^4.0.0",
"zod": "^3.23.8",
},
@ -220,12 +222,24 @@
"typescript": "^5.3.0",
},
},
"libs/core/handler-registry": {
"name": "@stock-bot/handler-registry",
"version": "1.0.0",
"dependencies": {
"@stock-bot/types": "workspace:*",
},
"devDependencies": {
"@types/bun": "*",
"typescript": "*",
},
},
"libs/core/handlers": {
"name": "@stock-bot/handlers",
"version": "1.0.0",
"dependencies": {
"@stock-bot/cache": "workspace:*",
"@stock-bot/config": "workspace:*",
"@stock-bot/handler-registry": "workspace:*",
"@stock-bot/logger": "workspace:*",
"@stock-bot/types": "workspace:*",
"@stock-bot/utils": "workspace:*",
@ -257,7 +271,7 @@
"version": "1.0.0",
"dependencies": {
"@stock-bot/cache": "*",
"@stock-bot/handlers": "*",
"@stock-bot/handler-registry": "*",
"@stock-bot/logger": "*",
"@stock-bot/types": "*",
"bullmq": "^5.0.0",
@ -820,6 +834,8 @@
"@stock-bot/event-bus": ["@stock-bot/event-bus@workspace:libs/core/event-bus"],
"@stock-bot/handler-registry": ["@stock-bot/handler-registry@workspace:libs/core/handler-registry"],
"@stock-bot/handlers": ["@stock-bot/handlers@workspace:libs/core/handlers"],
"@stock-bot/logger": ["@stock-bot/logger@workspace:libs/core/logger"],
@ -2144,7 +2160,7 @@
"side-channel-weakmap": ["side-channel-weakmap@1.0.2", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3", "side-channel-map": "^1.0.1" } }, "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A=="],
"signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
"signal-exit": ["signal-exit@4.1.0", "", {}, "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw=="],
"simple-concat": ["simple-concat@1.0.1", "", {}, "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q=="],
@ -2364,7 +2380,7 @@
"word-wrap": ["word-wrap@1.2.5", "", {}, "sha512-BN22B5eaMMI9UMtjrGd5g5eCYPpCPDUy0FJXbYsaT5zYxjFOckS53SQDE3pWkVoWpHXVb3BrYcEN4Twa55B5cA=="],
"wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="],
"wrap-ansi": ["wrap-ansi@8.1.0", "", { "dependencies": { "ansi-styles": "^6.1.0", "string-width": "^5.0.1", "strip-ansi": "^7.0.1" } }, "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ=="],
"wrap-ansi-cjs": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="],
@ -2430,8 +2446,6 @@
"@isaacs/cliui/strip-ansi": ["strip-ansi@7.1.0", "", { "dependencies": { "ansi-regex": "^6.0.1" } }, "sha512-iq6eVVI64nQQTRYq2KtEg2d2uU7LElhTJwsH4YzIHZshxlgZms/wIc4VoDQTlG/IvVIrBKG06CrZnp0qv7hkcQ=="],
"@isaacs/cliui/wrap-ansi": ["wrap-ansi@8.1.0", "", { "dependencies": { "ansi-styles": "^6.1.0", "string-width": "^5.0.1", "strip-ansi": "^7.0.1" } }, "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ=="],
"@mongodb-js/oidc-plugin/express": ["express@4.21.2", "", { "dependencies": { "accepts": "~1.3.8", "array-flatten": "1.1.1", "body-parser": "1.20.3", "content-disposition": "0.5.4", "content-type": "~1.0.4", "cookie": "0.7.1", "cookie-signature": "1.0.6", "debug": "2.6.9", "depd": "2.0.0", "encodeurl": "~2.0.0", "escape-html": "~1.0.3", "etag": "~1.8.1", "finalhandler": "1.3.1", "fresh": "0.5.2", "http-errors": "2.0.0", "merge-descriptors": "1.0.3", "methods": "~1.1.2", "on-finished": "2.4.1", "parseurl": "~1.3.3", "path-to-regexp": "0.1.12", "proxy-addr": "~2.0.7", "qs": "6.13.0", "range-parser": "~1.2.1", "safe-buffer": "5.2.1", "send": "0.19.0", "serve-static": "1.16.2", "setprototypeof": "1.2.0", "statuses": "2.0.1", "type-is": "~1.6.18", "utils-merge": "1.0.1", "vary": "~1.1.2" } }, "sha512-28HqgMZAmih1Czt9ny7qr6ek2qddF4FclbMzwhCREB6OFfH+rXAnuNCwo1/wFvrtbgsQDb4kSbX9de9lFbrXnA=="],
"@pm2/agent/chalk": ["chalk@3.0.0", "", { "dependencies": { "ansi-styles": "^4.1.0", "supports-color": "^7.1.0" } }, "sha512-4D3B6Wf41KOYRFdszmDqMCGq5VV/uMAB273JILmO+3jAlh8X4qDtdtgCR3fxtbLEMzSx22QdhnDcJvu2u1fVwg=="],
@ -2450,6 +2464,8 @@
"@pm2/io/semver": ["semver@7.5.4", "", { "dependencies": { "lru-cache": "^6.0.0" }, "bin": { "semver": "bin/semver.js" } }, "sha512-1bCSESV6Pv+i21Hvpxp3Dx+pSD8lIPt8uVjRrxAUt/nbswYc+tK6Y2btiULjd4+fnq15PX+nqQDC7Oft7WkwcA=="],
"@pm2/io/signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
"@pm2/io/tslib": ["tslib@1.9.3", "", {}, "sha512-4krF8scpejhaOgqzBEcGM7yDIEfi0/8+8zDRZhNZZ2kjmHJ4hv3zCbQWxoJGz1iw5U0Jl0nma13xzHXcncMavQ=="],
"@pm2/js-api/async": ["async@2.6.4", "", { "dependencies": { "lodash": "^4.17.14" } }, "sha512-mzo5dfJYwAn29PeiJ0zvwTo04zj8HDJj0Mn8TD7sno7q12prdbnasKJHhkm2c1LgrhlJ0teaea8860oxi51mGA=="],
@ -2516,6 +2532,8 @@
"cli-tableau/chalk": ["chalk@3.0.0", "", { "dependencies": { "ansi-styles": "^4.1.0", "supports-color": "^7.1.0" } }, "sha512-4D3B6Wf41KOYRFdszmDqMCGq5VV/uMAB273JILmO+3jAlh8X4qDtdtgCR3fxtbLEMzSx22QdhnDcJvu2u1fVwg=="],
"cliui/wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="],
"compress-commons/is-stream": ["is-stream@2.0.1", "", {}, "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg=="],
"decompress-response/mimic-response": ["mimic-response@3.1.0", "", {}, "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ=="],
@ -2550,14 +2568,14 @@
"execa/is-stream": ["is-stream@3.0.0", "", {}, "sha512-LnQR4bZ9IADDRSkvpqMGvt/tEJWclzklNgSw48V5EAaAeDd6qGvN8ei6k5p0tvxSR171VmGyHuTiAOfxAbr8kA=="],
"execa/signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
"express/cookie": ["cookie@0.7.2", "", {}, "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w=="],
"express/mime-types": ["mime-types@3.0.1", "", { "dependencies": { "mime-db": "^1.54.0" } }, "sha512-xRc4oEhT6eaBpU1XF7AjpOFD+xQmXNB5OVKwp4tqCuBpHLS/ZbBDrc07mYTDqVMg6PfxUjjNp85O6Cd2Z/5HWA=="],
"fast-glob/glob-parent": ["glob-parent@5.1.2", "", { "dependencies": { "is-glob": "^4.0.1" } }, "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow=="],
"foreground-child/signal-exit": ["signal-exit@4.1.0", "", {}, "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw=="],
"get-uri/data-uri-to-buffer": ["data-uri-to-buffer@6.0.2", "", {}, "sha512-7hvf7/GW8e86rW0ptuwS3OcBGDjIi6SZva7hCyWC0yYry2cOPmLIjXAUHI6DK2HsnwJd9ifmt57i8eV2n4YNpw=="],
"glob/minimatch": ["minimatch@9.0.5", "", { "dependencies": { "brace-expansion": "^2.0.1" } }, "sha512-G6T0ZX48xgozx7587koeX9Ys2NYy6Gmv//P89sEte9V9whIapMNF4idKxnW2QtCcLiTWlb/wfCabAtAFWhhBow=="],
@ -2620,6 +2638,8 @@
"prebuild-install/tar-fs": ["tar-fs@2.1.3", "", { "dependencies": { "chownr": "^1.1.1", "mkdirp-classic": "^0.5.2", "pump": "^3.0.0", "tar-stream": "^2.1.4" } }, "sha512-090nwYJDmlhwFwEW3QQl+vaNnxsO2yVsd45eTKRBzSzu+hlb1w2K9inVq5b0ngXuLVqQ4ApvsUHHnu/zQNkWAg=="],
"proper-lockfile/signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
"protobufjs/@types/node": ["@types/node@22.15.32", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-3jigKqgSjsH6gYZv2nEsqdXfZqIFGAV36XYYjf9KGZ3PSG+IhLecqPnI310RvjutyMwifE2hhhNEklOUrvx/wA=="],
"proxy-agent/lru-cache": ["lru-cache@7.18.3", "", {}, "sha512-jumlc0BIUrS3qJGgIkWZsyfAM7NCWiBcCDhnd+3NNM5KbBmLTgHVfWBcg6W+rLUsIpzpERPsvwUP7CckAQSOoA=="],
@ -2648,6 +2668,12 @@
"win-export-certificate-and-key/node-addon-api": ["node-addon-api@3.2.1", "", {}, "sha512-mmcei9JghVNDYydghQmeDX8KoAm0FAiYyIcUt/N4nhyAipB17pllZQDOJD2fotxABnt4Mdz+dKTO7eftLg4d0A=="],
"wrap-ansi/ansi-styles": ["ansi-styles@6.2.1", "", {}, "sha512-bN798gFfQX+viw3R7yrGWRqnrN2oRkEkUjjl4JNn4E8GxxbjtG3FbrEIIY3l8/hrwUwIeCZvi4QuOTP4MErVug=="],
"wrap-ansi/string-width": ["string-width@5.1.2", "", { "dependencies": { "eastasianwidth": "^0.2.0", "emoji-regex": "^9.2.2", "strip-ansi": "^7.0.1" } }, "sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA=="],
"wrap-ansi/strip-ansi": ["strip-ansi@7.1.0", "", { "dependencies": { "ansi-regex": "^6.0.1" } }, "sha512-iq6eVVI64nQQTRYq2KtEg2d2uU7LElhTJwsH4YzIHZshxlgZms/wIc4VoDQTlG/IvVIrBKG06CrZnp0qv7hkcQ=="],
"yauzl/buffer-crc32": ["buffer-crc32@0.2.13", "", {}, "sha512-VO9Ht/+p3SN7SKWqcrgEzjGbRSJYTx+Q1pTQC0wrWqHx0vpJraQ6GtHx8tvcg1rlK1byhU5gccxgOgj7B0TDkQ=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
@ -2660,8 +2686,6 @@
"@isaacs/cliui/strip-ansi/ansi-regex": ["ansi-regex@6.1.0", "", {}, "sha512-7HSX4QQb4CspciLpVFwyRe79O3xsIZDDLER21kERQ71oaPodF8jL725AgJMFAYbooIqolJoRLuM81SpeUkpkvA=="],
"@isaacs/cliui/wrap-ansi/ansi-styles": ["ansi-styles@6.2.1", "", {}, "sha512-bN798gFfQX+viw3R7yrGWRqnrN2oRkEkUjjl4JNn4E8GxxbjtG3FbrEIIY3l8/hrwUwIeCZvi4QuOTP4MErVug=="],
"@mongodb-js/oidc-plugin/express/accepts": ["accepts@1.3.8", "", { "dependencies": { "mime-types": "~2.1.34", "negotiator": "0.6.3" } }, "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw=="],
"@mongodb-js/oidc-plugin/express/body-parser": ["body-parser@1.20.3", "", { "dependencies": { "bytes": "3.1.2", "content-type": "~1.0.5", "debug": "2.6.9", "depd": "2.0.0", "destroy": "1.2.0", "http-errors": "2.0.0", "iconv-lite": "0.4.24", "on-finished": "2.4.1", "qs": "6.13.0", "raw-body": "2.5.2", "type-is": "~1.6.18", "unpipe": "1.0.0" } }, "sha512-7rAxByjUMqQ3/bHJy7D6OGXvx/MMc4IqBn/X0fcM1QUcAItpZrBEYhWGem+tzXH90c+G01ypMcYJBO9Y30203g=="],
@ -2880,12 +2904,18 @@
"run-applescript/execa/onetime": ["onetime@5.1.2", "", { "dependencies": { "mimic-fn": "^2.1.0" } }, "sha512-kbpaSSGJTWdAY5KPVeMOKXSrPtr8C8C7wodJbcsd51jRnmD+GZu8Y0VoU6Dm5Z4vWr0Ig/1NKuWRKf7j5aaYSg=="],
"run-applescript/execa/signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
"run-applescript/execa/strip-final-newline": ["strip-final-newline@2.0.0", "", {}, "sha512-BrpvfNAE3dcvq7ll3xVumzjKjZQ5tI1sEUIKr3Uoks0XUl45St3FlatVqef9prk4jRDzhW6WZg+3bk93y6pLjA=="],
"send/mime-types/mime-db": ["mime-db@1.54.0", "", {}, "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ=="],
"type-is/mime-types/mime-db": ["mime-db@1.54.0", "", {}, "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ=="],
"wrap-ansi/string-width/emoji-regex": ["emoji-regex@9.2.2", "", {}, "sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg=="],
"wrap-ansi/strip-ansi/ansi-regex": ["ansi-regex@6.1.0", "", {}, "sha512-7HSX4QQb4CspciLpVFwyRe79O3xsIZDDLER21kERQ71oaPodF8jL725AgJMFAYbooIqolJoRLuM81SpeUkpkvA=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
"@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],

1
libs/core/cache/src/namespaced-cache.ts vendored
View file

@ -57,7 +57,6 @@ export class NamespacedCache implements CacheProvider {
}
}
getStats() {
return this.cache.getStats();
}

2
libs/core/config/src/config-manager.ts
View file

@ -1,9 +1,9 @@
import { join } from 'path';
import { z } from 'zod';
import { getLogger } from '@stock-bot/logger';
import { EnvLoader } from './loaders/env.loader';
import { FileLoader } from './loaders/file.loader';
import { ConfigError, ConfigValidationError } from './errors';
import { getLogger } from '@stock-bot/logger';
import type {
ConfigLoader,
ConfigManagerOptions,

5
libs/core/config/src/index.ts
View file

@ -1,10 +1,10 @@
// Import necessary types
import { z } from 'zod';
import { EnvLoader } from './loaders/env.loader';
import { FileLoader } from './loaders/file.loader';
import { ConfigManager } from './config-manager';
import type { BaseAppConfig } from './schemas';
import { baseAppSchema } from './schemas';
import { z } from 'zod';
// Legacy singleton instance for backward compatibility
let configInstance: ConfigManager<BaseAppConfig> | null = null;
@ -56,7 +56,6 @@ function loadCriticalEnvVarsSync(): void {
// Load critical env vars immediately
loadCriticalEnvVarsSync();
/**
* Initialize configuration for a service in a monorepo.
* Automatically loads configs from:
@ -121,8 +120,6 @@ export function getLogConfig() {
return getConfig().log;
}
export function getQueueConfig() {
return getConfig().queue;
}

2
libs/core/config/src/schemas/__tests__/unified-app.test.ts
View file

@ -1,5 +1,5 @@
import { describe, expect, it } from 'bun:test';
import { unifiedAppSchema, toUnifiedConfig, getStandardServiceName } from '../unified-app.schema';
import { getStandardServiceName, toUnifiedConfig, unifiedAppSchema } from '../unified-app.schema';
describe('UnifiedAppConfig', () => {
describe('getStandardServiceName', () => {

30
libs/core/config/src/schemas/base-app.schema.ts
View file

@ -1,19 +1,19 @@
import { z } from 'zod';
import { environmentSchema } from './base.schema';
import {
postgresConfigSchema,
dragonflyConfigSchema,
mongodbConfigSchema,
postgresConfigSchema,
questdbConfigSchema,
dragonflyConfigSchema
} from './database.schema';
import {
serviceConfigSchema,
loggingConfigSchema,
queueConfigSchema,
httpConfigSchema,
webshareConfigSchema,
browserConfigSchema,
proxyConfigSchema
httpConfigSchema,
loggingConfigSchema,
proxyConfigSchema,
queueConfigSchema,
serviceConfigSchema,
webshareConfigSchema,
} from './service.schema';
/**
@ -32,12 +32,14 @@ export const baseAppSchema = z.object({
log: loggingConfigSchema,
// Database configuration - apps can choose which databases they need
database: z.object({
postgres: postgresConfigSchema.optional(),
mongodb: mongodbConfigSchema.optional(),
questdb: questdbConfigSchema.optional(),
dragonfly: dragonflyConfigSchema.optional(),
}).optional(),
database: z
.object({
postgres: postgresConfigSchema.optional(),
mongodb: mongodbConfigSchema.optional(),
questdb: questdbConfigSchema.optional(),
dragonfly: dragonflyConfigSchema.optional(),
})
.optional(),
// Redis configuration (used for cache and queue)
redis: dragonflyConfigSchema.optional(),

1
libs/core/config/src/schemas/index.ts
View file

@ -15,4 +15,3 @@ export type { BaseAppConfig } from './base-app.schema';
// Export unified schema for standardized configuration
export { unifiedAppSchema, toUnifiedConfig, getStandardServiceName } from './unified-app.schema';
export type { UnifiedAppConfig } from './unified-app.schema';

10
libs/core/config/src/schemas/service.schema.ts
View file

@ -100,8 +100,10 @@ export const proxyConfigSchema = z.object({
enabled: z.boolean().default(false),
cachePrefix: z.string().default('proxy:'),
ttl: z.number().default(3600),
webshare: z.object({
apiKey: z.string(),
apiUrl: z.string().default('https://proxy.webshare.io/api/v2/'),
}).optional(),
webshare: z
.object({
apiKey: z.string(),
apiUrl: z.string().default('https://proxy.webshare.io/api/v2/'),
})
.optional(),
});

96
libs/core/config/src/schemas/unified-app.schema.ts
View file

@ -1,62 +1,67 @@
import { z } from 'zod';
import { baseAppSchema } from './base-app.schema';
import {
postgresConfigSchema,
dragonflyConfigSchema,
mongodbConfigSchema,
postgresConfigSchema,
questdbConfigSchema,
dragonflyConfigSchema
} from './database.schema';
/**
* Unified application configuration schema that provides both nested and flat access
* to database configurations for backward compatibility while maintaining a clean structure
*/
export const unifiedAppSchema = baseAppSchema.extend({
// Flat database configs for DI system (these take precedence)
redis: dragonflyConfigSchema.optional(),
mongodb: mongodbConfigSchema.optional(),
postgres: postgresConfigSchema.optional(),
questdb: questdbConfigSchema.optional(),
}).transform((data) => {
// Ensure service.serviceName is set from service.name if not provided
if (data.service && !data.service.serviceName) {
data.service.serviceName = data.service.name.replace(/([A-Z])/g, '-$1').toLowerCase().replace(/^-/, '');
}
export const unifiedAppSchema = baseAppSchema
.extend({
// Flat database configs for DI system (these take precedence)
redis: dragonflyConfigSchema.optional(),
mongodb: mongodbConfigSchema.optional(),
postgres: postgresConfigSchema.optional(),
questdb: questdbConfigSchema.optional(),
})
.transform(data => {
// Ensure service.serviceName is set from service.name if not provided
if (data.service && !data.service.serviceName) {
data.service.serviceName = data.service.name
.replace(/([A-Z])/g, '-$1')
.toLowerCase()
.replace(/^-/, '');
}
// If flat configs exist, ensure they're also in the nested database object
if (data.redis || data.mongodb || data.postgres || data.questdb) {
data.database = {
...data.database,
dragonfly: data.redis || data.database?.dragonfly,
mongodb: data.mongodb || data.database?.mongodb,
postgres: data.postgres || data.database?.postgres,
questdb: data.questdb || data.database?.questdb,
};
}
// If flat configs exist, ensure they're also in the nested database object
if (data.redis || data.mongodb || data.postgres || data.questdb) {
data.database = {
...data.database,
dragonfly: data.redis || data.database?.dragonfly,
mongodb: data.mongodb || data.database?.mongodb,
postgres: data.postgres || data.database?.postgres,
questdb: data.questdb || data.database?.questdb,
};
}
// If nested configs exist but flat ones don't, copy them to flat structure
if (data.database) {
if (data.database.dragonfly && !data.redis) {
data.redis = data.database.dragonfly;
}
if (data.database.mongodb && !data.mongodb) {
data.mongodb = data.database.mongodb;
}
if (data.database.postgres && !data.postgres) {
data.postgres = data.database.postgres;
}
if (data.database.questdb && !data.questdb) {
// Handle the ilpPort -> influxPort mapping for DI system
const questdbConfig = { ...data.database.questdb };
if ('ilpPort' in questdbConfig && !('influxPort' in questdbConfig)) {
(questdbConfig as any).influxPort = questdbConfig.ilpPort;
// If nested configs exist but flat ones don't, copy them to flat structure
if (data.database) {
if (data.database.dragonfly && !data.redis) {
data.redis = data.database.dragonfly;
}
if (data.database.mongodb && !data.mongodb) {
data.mongodb = data.database.mongodb;
}
if (data.database.postgres && !data.postgres) {
data.postgres = data.database.postgres;
}
if (data.database.questdb && !data.questdb) {
// Handle the ilpPort -> influxPort mapping for DI system
const questdbConfig = { ...data.database.questdb };
if ('ilpPort' in questdbConfig && !('influxPort' in questdbConfig)) {
(questdbConfig as any).influxPort = questdbConfig.ilpPort;
}
data.questdb = questdbConfig;
}
data.questdb = questdbConfig;
}
}
return data;
});
return data;
});
export type UnifiedAppConfig = z.infer<typeof unifiedAppSchema>;
@ -72,5 +77,8 @@ export function toUnifiedConfig(config: any): UnifiedAppConfig {
*/
export function getStandardServiceName(serviceName: string): string {
// Convert camelCase to kebab-case
return serviceName.replace(/([A-Z])/g, '-$1').toLowerCase().replace(/^-/, '');
return serviceName
.replace(/([A-Z])/g, '-$1')
.toLowerCase()
.replace(/^-/, '');
}

2
libs/core/di/package.json
View file

@ -20,6 +20,8 @@
"@stock-bot/queue": "workspace:*",
"@stock-bot/shutdown": "workspace:*",
"@stock-bot/handlers": "workspace:*",
"@stock-bot/handler-registry": "workspace:*",
"glob": "^10.0.0",
"zod": "^3.23.8",
"hono": "^4.0.0",
"awilix": "^12.0.5"

8
libs/core/di/src/awilix-container.ts
View file

@ -3,16 +3,16 @@
* Creates a decoupled, reusable dependency injection container
*/
import { type AwilixContainer } from 'awilix';
import type { Browser } from '@stock-bot/browser';
import type { CacheProvider } from '@stock-bot/cache';
import type { IServiceContainer } from '@stock-bot/types';
import type { Logger } from '@stock-bot/logger';
import type { MongoDBClient } from '@stock-bot/mongodb';
import type { PostgreSQLClient } from '@stock-bot/postgres';
import type { ProxyManager } from '@stock-bot/proxy';
import type { QuestDBClient } from '@stock-bot/questdb';
import type { QueueManager } from '@stock-bot/queue';
import { type AwilixContainer } from 'awilix';
import type { IServiceContainer } from '@stock-bot/types';
import type { AppConfig } from './config/schemas';
// Re-export for backward compatibility
@ -41,8 +41,6 @@ export interface ServiceDefinitions {
serviceContainer: IServiceContainer;
}
// Export typed container
export type ServiceContainer = AwilixContainer<ServiceDefinitions>;
export type ServiceCradle = ServiceDefinitions;
@ -59,5 +57,3 @@ export interface ServiceContainerOptions {
enableBrowser?: boolean;
enableProxy?: boolean;
}

16
libs/core/di/src/config/schemas/index.ts
View file

@ -1,9 +1,9 @@
import { z } from 'zod';
import { redisConfigSchema } from './redis.schema';
import { mongodbConfigSchema } from './mongodb.schema';
import { postgresConfigSchema } from './postgres.schema';
import { questdbConfigSchema } from './questdb.schema';
import { proxyConfigSchema, browserConfigSchema, queueConfigSchema } from './service.schema';
import { redisConfigSchema } from './redis.schema';
import { browserConfigSchema, proxyConfigSchema, queueConfigSchema } from './service.schema';
export const appConfigSchema = z.object({
redis: redisConfigSchema,
@ -13,11 +13,13 @@ export const appConfigSchema = z.object({
proxy: proxyConfigSchema.optional(),
browser: browserConfigSchema.optional(),
queue: queueConfigSchema.optional(),
service: z.object({
name: z.string(),
serviceName: z.string().optional(), // Standard kebab-case service name
port: z.number().optional(),
}).optional(),
service: z
.object({
name: z.string(),
serviceName: z.string().optional(), // Standard kebab-case service name
port: z.number().optional(),
})
.optional(),
});
export type AppConfig = z.infer<typeof appConfigSchema>;

35
libs/core/di/src/config/schemas/service.schema.ts
View file

@ -4,10 +4,12 @@ export const proxyConfigSchema = z.object({
enabled: z.boolean().default(false),
cachePrefix: z.string().optional().default('proxy:'),
ttl: z.number().optional().default(3600),
webshare: z.object({
apiKey: z.string(),
apiUrl: z.string().default('https://proxy.webshare.io/api/v2/'),
}).optional(),
webshare: z
.object({
apiKey: z.string(),
apiUrl: z.string().default('https://proxy.webshare.io/api/v2/'),
})
.optional(),
});
export const browserConfigSchema = z.object({
@ -21,16 +23,21 @@ export const queueConfigSchema = z.object({
concurrency: z.number().optional().default(1),
enableScheduledJobs: z.boolean().optional().default(true),
delayWorkerStart: z.boolean().optional().default(false),
defaultJobOptions: z.object({
attempts: z.number().default(3),
backoff: z.object({
type: z.enum(['exponential', 'fixed']).default('exponential'),
delay: z.number().default(1000),
}).default({}),
removeOnComplete: z.number().default(100),
removeOnFail: z.number().default(50),
timeout: z.number().optional(),
}).optional().default({}),
defaultJobOptions: z
.object({
attempts: z.number().default(3),
backoff: z
.object({
type: z.enum(['exponential', 'fixed']).default('exponential'),
delay: z.number().default(1000),
})
.default({}),
removeOnComplete: z.number().default(100),
removeOnFail: z.number().default(50),
timeout: z.number().optional(),
})
.optional()
.default({}),
});
export type ProxyConfig = z.infer<typeof proxyConfigSchema>;

144
libs/core/di/src/container/builder.ts
View file

@ -1,15 +1,17 @@
import { createContainer, InjectionMode, asFunction, type AwilixContainer } from 'awilix';
import { asClass, asFunction, createContainer, InjectionMode, type AwilixContainer } from 'awilix';
import type { BaseAppConfig as StockBotAppConfig, UnifiedAppConfig } from '@stock-bot/config';
import { appConfigSchema, type AppConfig } from '../config/schemas';
import { toUnifiedConfig } from '@stock-bot/config';
import { HandlerRegistry } from '@stock-bot/handler-registry';
import { appConfigSchema, type AppConfig } from '../config/schemas';
import {
registerCoreServices,
registerApplicationServices,
registerCacheServices,
registerCoreServices,
registerDatabaseServices,
registerApplicationServices
} from '../registrations';
import { HandlerScanner } from '../scanner';
import { ServiceLifecycleManager } from '../utils/lifecycle';
import type { ServiceDefinitions, ContainerBuildOptions } from './types';
import type { ContainerBuildOptions, ServiceDefinitions } from './types';
export class ServiceContainerBuilder {
private config: Partial<AppConfig> = {};
@ -38,7 +40,10 @@ export class ServiceContainerBuilder {
return this;
}
enableService(service: keyof Omit<ContainerBuildOptions, 'skipInitialization' | 'initializationTimeout'>, enabled = true): this {
enableService(
service: keyof Omit<ContainerBuildOptions, 'skipInitialization' | 'initializationTimeout'>,
enabled = true
): this {
this.options[service] = enabled;
return this;
}
@ -77,17 +82,19 @@ export class ServiceContainerBuilder {
private applyServiceOptions(config: Partial<AppConfig>): AppConfig {
// Ensure questdb config has the right field names for DI
const questdbConfig = config.questdb ? {
...config.questdb,
influxPort: (config.questdb as any).influxPort || (config.questdb as any).ilpPort || 9009,
} : {
enabled: true,
host: 'localhost',
httpPort: 9000,
pgPort: 8812,
influxPort: 9009,
database: 'questdb',
};
const questdbConfig = config.questdb
? {
...config.questdb,
influxPort: (config.questdb as any).influxPort || (config.questdb as any).ilpPort || 9009,
}
: {
enabled: true,
host: 'localhost',
httpPort: 9000,
pgPort: 8812,
influxPort: 9009,
database: 'questdb',
};
return {
redis: config.redis || {
@ -110,26 +117,41 @@ export class ServiceContainerBuilder {
password: 'postgres',
},
questdb: this.options.enableQuestDB ? questdbConfig : undefined,
proxy: this.options.enableProxy ? (config.proxy || { enabled: false, cachePrefix: 'proxy:', ttl: 3600 }) : undefined,
browser: this.options.enableBrowser ? (config.browser || { headless: true, timeout: 30000 }) : undefined,
queue: this.options.enableQueue ? (config.queue || {
enabled: true,
workers: 1,
concurrency: 1,
enableScheduledJobs: true,
delayWorkerStart: false,
defaultJobOptions: {
attempts: 3,
backoff: { type: 'exponential' as const, delay: 1000 },
removeOnComplete: 100,
removeOnFail: 50,
}
}) : undefined,
proxy: this.options.enableProxy
? config.proxy || { enabled: false, cachePrefix: 'proxy:', ttl: 3600 }
: undefined,
browser: this.options.enableBrowser
? config.browser || { headless: true, timeout: 30000 }
: undefined,
queue: this.options.enableQueue
? config.queue || {
enabled: true,
workers: 1,
concurrency: 1,
enableScheduledJobs: true,
delayWorkerStart: false,
defaultJobOptions: {
attempts: 3,
backoff: { type: 'exponential' as const, delay: 1000 },
removeOnComplete: 100,
removeOnFail: 50,
},
}
: undefined,
service: config.service,
};
}
private registerServices(container: AwilixContainer<ServiceDefinitions>, config: AppConfig): void {
private registerServices(
container: AwilixContainer<ServiceDefinitions>,
config: AppConfig
): void {
// Register handler infrastructure first
container.register({
handlerRegistry: asClass(HandlerRegistry).singleton(),
handlerScanner: asClass(HandlerScanner).singleton(),
});
registerCoreServices(container, config);
registerCacheServices(container, config);
registerDatabaseServices(container, config);
@ -137,34 +159,46 @@ export class ServiceContainerBuilder {
// Register service container aggregate
container.register({
serviceContainer: asFunction(({
config: _config, logger, cache, globalCache, proxyManager, browser,
queueManager, mongoClient, postgresClient, questdbClient
}) => ({
logger,
cache,
globalCache,
proxy: proxyManager, // Map proxyManager to proxy
browser,
queue: queueManager, // Map queueManager to queue
mongodb: mongoClient, // Map mongoClient to mongodb
postgres: postgresClient, // Map postgresClient to postgres
questdb: questdbClient, // Map questdbClient to questdb
})).singleton(),
serviceContainer: asFunction(
({
config: _config,
logger,
cache,
globalCache,
proxyManager,
browser,
queueManager,
mongoClient,
postgresClient,
questdbClient,
}) => ({
logger,
cache,
globalCache,
proxy: proxyManager, // Map proxyManager to proxy
browser,
queue: queueManager, // Map queueManager to queue
mongodb: mongoClient, // Map mongoClient to mongodb
postgres: postgresClient, // Map postgresClient to postgres
questdb: questdbClient, // Map questdbClient to questdb
})
).singleton(),
});
}
private transformStockBotConfig(config: UnifiedAppConfig): Partial<AppConfig> {
// Unified config already has flat structure, just extract what we need
// Handle questdb field name mapping
const questdb = config.questdb ? {
enabled: config.questdb.enabled || true,
host: config.questdb.host || 'localhost',
httpPort: config.questdb.httpPort || 9000,
pgPort: config.questdb.pgPort || 8812,
influxPort: (config.questdb as any).influxPort || (config.questdb as any).ilpPort || 9009,
database: config.questdb.database || 'questdb',
} : undefined;
const questdb = config.questdb
? {
enabled: config.questdb.enabled || true,
host: config.questdb.host || 'localhost',
httpPort: config.questdb.httpPort || 9000,
pgPort: config.questdb.pgPort || 8812,
influxPort: (config.questdb as any).influxPort || (config.questdb as any).ilpPort || 9009,
database: config.questdb.database || 'questdb',
}
: undefined;
return {
redis: config.redis,

8
libs/core/di/src/container/types.ts
View file

@ -1,19 +1,25 @@
import type { Browser } from '@stock-bot/browser';
import type { CacheProvider } from '@stock-bot/cache';
import type { IServiceContainer } from '@stock-bot/types';
import type { HandlerRegistry } from '@stock-bot/handler-registry';
import type { Logger } from '@stock-bot/logger';
import type { MongoDBClient } from '@stock-bot/mongodb';
import type { PostgreSQLClient } from '@stock-bot/postgres';
import type { ProxyManager } from '@stock-bot/proxy';
import type { QuestDBClient } from '@stock-bot/questdb';
import type { SmartQueueManager } from '@stock-bot/queue';
import type { IServiceContainer } from '@stock-bot/types';
import type { AppConfig } from '../config/schemas';
import type { HandlerScanner } from '../scanner';
export interface ServiceDefinitions {
// Configuration
config: AppConfig;
logger: Logger;
// Handler infrastructure
handlerRegistry: HandlerRegistry;
handlerScanner: HandlerScanner;
// Core services
cache: CacheProvider | null;
globalCache: CacheProvider | null;

17
libs/core/di/src/factories/cache.factory.ts
View file

@ -3,10 +3,7 @@ import { NamespacedCache, type CacheProvider } from '@stock-bot/cache';
import type { ServiceDefinitions } from '../container/types';
export class CacheFactory {
static createNamespacedCache(
baseCache: CacheProvider,
namespace: string
): NamespacedCache {
static createNamespacedCache(baseCache: CacheProvider, namespace: string): NamespacedCache {
return new NamespacedCache(baseCache, namespace);
}
@ -15,7 +12,9 @@ export class CacheFactory {
serviceName: string
): CacheProvider | null {
const baseCache = container.cradle.cache;
if (!baseCache) {return null;}
if (!baseCache) {
return null;
}
return this.createNamespacedCache(baseCache, serviceName);
}
@ -25,7 +24,9 @@ export class CacheFactory {
handlerName: string
): CacheProvider | null {
const baseCache = container.cradle.cache;
if (!baseCache) {return null;}
if (!baseCache) {
return null;
}
return this.createNamespacedCache(baseCache, `handler:${handlerName}`);
}
@ -35,7 +36,9 @@ export class CacheFactory {
prefix: string
): CacheProvider | null {
const baseCache = container.cradle.cache;
if (!baseCache) {return null;}
if (!baseCache) {
return null;
}
// Remove 'cache:' prefix if already included
const cleanPrefix = prefix.replace(/^cache:/, '');

4
libs/core/di/src/index.ts
View file

@ -33,3 +33,7 @@ export {
type ServiceApplicationConfig,
type ServiceLifecycleHooks,
} from './service-application';
// Handler scanner
export { HandlerScanner } from './scanner';
export type { HandlerScannerOptions } from './scanner';

30
libs/core/di/src/registrations/cache.registration.ts
View file

@ -14,12 +14,16 @@ export function registerCacheServices(
const serviceName = config.service?.serviceName || config.service?.name || 'unknown';
// Create service-specific cache that uses the service's Redis DB
return createServiceCache(serviceName, {
host: config.redis.host,
port: config.redis.port,
password: config.redis.password,
db: config.redis.db, // This will be overridden by ServiceCache
}, { logger });
return createServiceCache(
serviceName,
{
host: config.redis.host,
port: config.redis.port,
password: config.redis.password,
db: config.redis.db, // This will be overridden by ServiceCache
},
{ logger }
);
}).singleton(),
// Also provide global cache for shared data
@ -27,11 +31,15 @@ export function registerCacheServices(
const { createServiceCache } = require('@stock-bot/queue');
const serviceName = config.service?.serviceName || config.service?.name || 'unknown';
return createServiceCache(serviceName, {
host: config.redis.host,
port: config.redis.port,
password: config.redis.password,
}, { global: true, logger });
return createServiceCache(
serviceName,
{
host: config.redis.host,
port: config.redis.port,
password: config.redis.password,
},
{ global: true, logger }
);
}).singleton(),
});
} else {

6
libs/core/di/src/registrations/database.registration.ts
View file

@ -1,7 +1,7 @@
import { asFunction, asValue, type AwilixContainer } from 'awilix';
import { MongoDBClient } from '@stock-bot/mongodb';
import { PostgreSQLClient } from '@stock-bot/postgres';
import { QuestDBClient } from '@stock-bot/questdb';
import { asFunction, asValue, type AwilixContainer } from 'awilix';
import type { AppConfig } from '../config/schemas';
import type { ServiceDefinitions } from '../container/types';
@ -14,7 +14,9 @@ export function registerDatabaseServices(
container.register({
mongoClient: asFunction(({ logger }) => {
// Parse MongoDB URI to extract components
const uriMatch = config.mongodb.uri.match(/mongodb:\/\/(?:([^:]+):([^@]+)@)?([^:/]+):(\d+)\/([^?]+)(?:\?authSource=(.+))?/);
const uriMatch = config.mongodb.uri.match(
/mongodb:\/\/(?:([^:]+):([^@]+)@)?([^:/]+):(\d+)\/([^?]+)(?:\?authSource=(.+))?/
);
const mongoConfig = {
host: uriMatch?.[3] || 'localhost',
port: parseInt(uriMatch?.[4] || '27017'),

4
libs/core/di/src/registrations/service.registration.ts
View file

@ -60,7 +60,7 @@ export function registerApplicationServices(
// Queue Manager
if (config.queue?.enabled && config.redis.enabled) {
container.register({
queueManager: asFunction(({ logger }) => {
queueManager: asFunction(({ logger, handlerRegistry }) => {
const { SmartQueueManager } = require('@stock-bot/queue');
const queueConfig = {
serviceName: config.service?.serviceName || config.service?.name || 'unknown',
@ -79,7 +79,7 @@ export function registerApplicationServices(
delayWorkerStart: config.queue!.delayWorkerStart ?? false,
autoDiscoverHandlers: true,
};
return new SmartQueueManager(queueConfig, logger);
return new SmartQueueManager(queueConfig, handlerRegistry, logger);
}).singleton(),
});
} else {

201
libs/core/di/src/scanner/handler-scanner.ts Normal file
View file

@ -0,0 +1,201 @@
/**
* Handler Scanner
* Discovers and registers handlers with the DI container
*/
import { asClass, type AwilixContainer } from 'awilix';
import { glob } from 'glob';
import type {
HandlerConfiguration,
HandlerMetadata,
HandlerRegistry,
} from '@stock-bot/handler-registry';
import { createJobHandler } from '@stock-bot/handlers';
import { getLogger } from '@stock-bot/logger';
import type { ExecutionContext, IHandler } from '@stock-bot/types';
export interface HandlerScannerOptions {
serviceName?: string;
autoRegister?: boolean;
patterns?: string[];
}
export class HandlerScanner {
private logger = getLogger('handler-scanner');
private discoveredHandlers = new Map<string, any>();
constructor(
private registry: HandlerRegistry,
private container: AwilixContainer,
private options: HandlerScannerOptions = {}
) {}
/**
* Scan for handlers matching the given patterns
*/
async scanHandlers(patterns: string[] = this.options.patterns || []): Promise<void> {
this.logger.info('Starting handler scan', { patterns });
for (const pattern of patterns) {
const files = await glob(pattern, { absolute: true });
this.logger.debug(`Found ${files.length} files for pattern: ${pattern}`);
for (const file of files) {
try {
await this.scanFile(file);
} catch (error) {
this.logger.error('Failed to scan file', { file, error });
}
}
}
this.logger.info('Handler scan complete', {
discovered: this.discoveredHandlers.size,
patterns,
});
}
/**
* Scan a single file for handlers
*/
private async scanFile(filePath: string): Promise<void> {
try {
const module = await import(filePath);
this.registerHandlersFromModule(module, filePath);
} catch (error) {
this.logger.error('Failed to import module', { filePath, error });
}
}
/**
* Register handlers found in a module
*/
private registerHandlersFromModule(module: any, filePath: string): void {
for (const [exportName, exported] of Object.entries(module)) {
if (this.isHandler(exported)) {
this.registerHandler(exported, exportName, filePath);
}
}
}
/**
* Check if an exported value is a handler
*/
private isHandler(exported: any): boolean {
if (typeof exported !== 'function') return false;
// Check for handler metadata added by decorators
const hasHandlerName = !!(exported as any).__handlerName;
const hasOperations = Array.isArray((exported as any).__operations);
return hasHandlerName && hasOperations;
}
/**
* Register a handler with the registry and DI container
*/
private registerHandler(HandlerClass: any, exportName: string, filePath: string): void {
const handlerName = HandlerClass.__handlerName;
const operations = HandlerClass.__operations || [];
const schedules = HandlerClass.__schedules || [];
const isDisabled = HandlerClass.__disabled || false;
if (isDisabled) {
this.logger.debug('Skipping disabled handler', { handlerName });
return;
}
// Build metadata
const metadata: HandlerMetadata = {
name: handlerName,
service: this.options.serviceName,
operations: operations.map((op: any) => ({
name: op.name,
method: op.method,
})),
schedules: schedules.map((schedule: any) => ({
operation: schedule.operation,
cronPattern: schedule.cronPattern,
priority: schedule.priority,
immediately: schedule.immediately,
description: schedule.description,
})),
};
// Build configuration with operation handlers
const operationHandlers: Record<string, any> = {};
for (const op of operations) {
operationHandlers[op.name] = createJobHandler(async payload => {
const handler = this.container.resolve<IHandler>(handlerName);
const context: ExecutionContext = {
type: 'queue',
metadata: { source: 'queue', timestamp: Date.now() },
};
return await handler.execute(op.name, payload, context);
});
}
const configuration: HandlerConfiguration = {
name: handlerName,
operations: operationHandlers,
scheduledJobs: schedules.map((schedule: any) => {
const operation = operations.find((op: any) => op.method === schedule.operation);
return {
type: `${handlerName}-${schedule.operation}`,
operation: operation?.name || schedule.operation,
cronPattern: schedule.cronPattern,
priority: schedule.priority || 5,
immediately: schedule.immediately || false,
description: schedule.description || `${handlerName} ${schedule.operation}`,
};
}),
};
// Register with registry
this.registry.register(metadata, configuration);
// Register with DI container if auto-register is enabled
if (this.options.autoRegister !== false) {
this.container.register({
[handlerName]: asClass(HandlerClass).singleton(),
});
}
// Track discovered handler
this.discoveredHandlers.set(handlerName, HandlerClass);
this.logger.info('Registered handler', {
handlerName,
exportName,
filePath,
operations: operations.length,
schedules: schedules.length,
service: this.options.serviceName,
});
}
/**
* Get all discovered handlers
*/
getDiscoveredHandlers(): Map<string, any> {
return new Map(this.discoveredHandlers);
}
/**
* Manually register a handler class
*/
registerHandlerClass(HandlerClass: any, options: { serviceName?: string } = {}): void {
const serviceName = options.serviceName || this.options.serviceName;
const originalServiceName = this.options.serviceName;
// Temporarily override service name if provided
if (serviceName) {
this.options.serviceName = serviceName;
}
this.registerHandler(HandlerClass, HandlerClass.name, 'manual');
// Restore original service name
this.options.serviceName = originalServiceName;
}
}

2
libs/core/di/src/scanner/index.ts Normal file
View file

@ -0,0 +1,2 @@
export { HandlerScanner } from './handler-scanner';
export type { HandlerScannerOptions } from './handler-scanner';

19
libs/core/di/src/service-application.ts
View file

@ -5,12 +5,14 @@
import { Hono } from 'hono';
import { cors } from 'hono/cors';
import { getLogger, setLoggerConfig, shutdownLoggers, type Logger } from '@stock-bot/logger';
import { Shutdown } from '@stock-bot/shutdown';
import type { BaseAppConfig, UnifiedAppConfig } from '@stock-bot/config';
import { toUnifiedConfig } from '@stock-bot/config';
import { getLogger, setLoggerConfig, shutdownLoggers, type Logger } from '@stock-bot/logger';
import { Shutdown } from '@stock-bot/shutdown';
import type { IServiceContainer } from '@stock-bot/types';
import type { ServiceContainer } from './awilix-container';
import type { HandlerRegistry } from '@stock-bot/handler-registry';
import type { ServiceDefinitions } from './container/types';
import type { AwilixContainer } from 'awilix';
/**
* Configuration for ServiceApplication
@ -71,7 +73,7 @@ export class ServiceApplication {
private hooks: ServiceLifecycleHooks;
private logger: Logger;
private container: ServiceContainer | null = null;
private container: AwilixContainer<ServiceDefinitions> | null = null;
private serviceContainer: IServiceContainer | null = null;
private app: Hono | null = null;
private server: ReturnType<typeof Bun.serve> | null = null;
@ -105,7 +107,7 @@ export class ServiceApplication {
// Initialize shutdown manager
this.shutdown = Shutdown.getInstance({
timeout: this.serviceConfig.shutdownTimeout
timeout: this.serviceConfig.shutdownTimeout,
});
}
@ -246,7 +248,7 @@ export class ServiceApplication {
* Start the service with full initialization
*/
async start(
containerFactory: (config: UnifiedAppConfig) => Promise<ServiceContainer>,
containerFactory: (config: UnifiedAppConfig) => Promise<AwilixContainer<ServiceDefinitions>>,
routeFactory: (container: IServiceContainer) => Hono,
handlerInitializer?: (container: IServiceContainer) => Promise<void>
): Promise<void> {
@ -257,7 +259,7 @@ export class ServiceApplication {
this.logger.debug('Creating DI container...');
// Config already has service name from constructor
this.container = await containerFactory(this.config);
this.serviceContainer = this.container.resolve('serviceContainer');
this.serviceContainer = this.container!.resolve('serviceContainer');
this.logger.info('DI container created and initialized');
// Call container ready hook
@ -311,7 +313,6 @@ export class ServiceApplication {
if (this.hooks.onStarted) {
await this.hooks.onStarted(port);
}
} catch (error) {
this.logger.error('DETAILED ERROR:', error);
this.logger.error('Failed to start service', {
@ -332,7 +333,7 @@ export class ServiceApplication {
}
this.logger.debug('Creating scheduled jobs from registered handlers...');
const { handlerRegistry } = await import('@stock-bot/handlers');
const handlerRegistry = this.container.resolve<HandlerRegistry>('handlerRegistry');
const allHandlers = handlerRegistry.getAllHandlersWithSchedule();
let totalScheduledJobs = 0;

7
libs/core/di/src/utils/lifecycle.ts
View file

@ -1,6 +1,6 @@
import type { AwilixContainer } from 'awilix';
import type { ServiceDefinitions } from '../container/types';
import { getLogger } from '@stock-bot/logger';
import type { ServiceDefinitions } from '../container/types';
interface ServiceWithLifecycle {
connect?: () => Promise<void>;
@ -35,7 +35,10 @@ export class ServiceLifecycleManager {
initPromises.push(
Promise.race([
initPromise,
this.createTimeoutPromise(timeout, `${name} initialization timed out after ${timeout}ms`),
this.createTimeoutPromise(
timeout,
`${name} initialization timed out after ${timeout}ms`
),
])
);
}

27
libs/core/handler-registry/package.json Normal file
View file

@ -0,0 +1,27 @@
{
"name": "@stock-bot/handler-registry",
"version": "1.0.0",
"type": "module",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
"exports": {
".": {
"import": "./dist/index.js",
"types": "./dist/index.d.ts"
}
},
"scripts": {
"build": "bun run build:clean && bun run build:tsc",
"build:clean": "rm -rf dist",
"build:tsc": "tsc",
"test": "bun test",
"clean": "rm -rf dist node_modules .turbo"
},
"dependencies": {
"@stock-bot/types": "workspace:*"
},
"devDependencies": {
"@types/bun": "*",
"typescript": "*"
}
}

14
libs/core/handler-registry/src/index.ts Normal file
View file

@ -0,0 +1,14 @@
/**
* Handler Registry Package
* Provides centralized handler registration without circular dependencies
*/
export { HandlerRegistry } from './registry';
export type {
HandlerMetadata,
OperationMetadata,
ScheduleMetadata,
HandlerConfiguration,
RegistryStats,
HandlerDiscoveryResult,
} from './types';

226
libs/core/handler-registry/src/registry.ts Normal file
View file

@ -0,0 +1,226 @@
/**
* Handler Registry Implementation
* Manages handler metadata and configuration without circular dependencies
*/
import type { JobHandler, ScheduledJob } from '@stock-bot/types';
import type {
HandlerConfiguration,
HandlerMetadata,
OperationMetadata,
RegistryStats,
ScheduleMetadata,
} from './types';
export class HandlerRegistry {
private handlers = new Map<string, HandlerMetadata>();
private configurations = new Map<string, HandlerConfiguration>();
private handlerServices = new Map<string, string>();
/**
* Register handler metadata
*/
registerMetadata(metadata: HandlerMetadata): void {
this.handlers.set(metadata.name, metadata);
if (metadata.service) {
this.handlerServices.set(metadata.name, metadata.service);
}
}
/**
* Register handler configuration with operation implementations
*/
registerConfiguration(config: HandlerConfiguration): void {
this.configurations.set(config.name, config);
}
/**
* Register both metadata and configuration
*/
register(metadata: HandlerMetadata, config: HandlerConfiguration): void {
this.registerMetadata(metadata);
this.registerConfiguration(config);
}
/**
* Get handler metadata
*/
getMetadata(handlerName: string): HandlerMetadata | undefined {
return this.handlers.get(handlerName);
}
/**
* Get handler configuration
*/
getConfiguration(handlerName: string): HandlerConfiguration | undefined {
return this.configurations.get(handlerName);
}
/**
* Get a specific operation handler
*/
getOperation(handlerName: string, operationName: string): JobHandler | undefined {
const config = this.configurations.get(handlerName);
return config?.operations[operationName];
}
/**
* Get all handler metadata
*/
getAllMetadata(): Map<string, HandlerMetadata> {
return new Map(this.handlers);
}
/**
* Get all handler names
*/
getHandlerNames(): string[] {
return Array.from(this.handlers.keys());
}
/**
* Check if a handler is registered
*/
hasHandler(handlerName: string): boolean {
return this.handlers.has(handlerName);
}
/**
* Get handlers for a specific service
*/
getServiceHandlers(serviceName: string): HandlerMetadata[] {
const handlers: HandlerMetadata[] = [];
for (const [handlerName, service] of this.handlerServices) {
if (service === serviceName) {
const metadata = this.handlers.get(handlerName);
if (metadata) {
handlers.push(metadata);
}
}
}
return handlers;
}
/**
* Set service ownership for a handler
*/
setHandlerService(handlerName: string, serviceName: string): void {
this.handlerServices.set(handlerName, serviceName);
// Update metadata if it exists
const metadata = this.handlers.get(handlerName);
if (metadata) {
metadata.service = serviceName;
}
}
/**
* Get the service that owns a handler
*/
getHandlerService(handlerName: string): string | undefined {
return this.handlerServices.get(handlerName);
}
/**
* Get scheduled jobs for a handler
*/
getScheduledJobs(handlerName: string): ScheduledJob[] {
const config = this.configurations.get(handlerName);
return config?.scheduledJobs || [];
}
/**
* Get all handlers with their scheduled jobs
*/
getAllHandlersWithSchedule(): Map<
string,
{ metadata: HandlerMetadata; scheduledJobs: ScheduledJob[] }
> {
const result = new Map<string, { metadata: HandlerMetadata; scheduledJobs: ScheduledJob[] }>();
for (const [name, metadata] of this.handlers) {
const config = this.configurations.get(name);
result.set(name, {
metadata,
scheduledJobs: config?.scheduledJobs || [],
});
}
return result;
}
/**
* Get registry statistics
*/
getStats(): RegistryStats {
let operationCount = 0;
let scheduledJobCount = 0;
const services = new Set<string>();
for (const metadata of this.handlers.values()) {
operationCount += metadata.operations.length;
scheduledJobCount += metadata.schedules?.length || 0;
if (metadata.service) {
services.add(metadata.service);
}
}
return {
handlers: this.handlers.size,
operations: operationCount,
scheduledJobs: scheduledJobCount,
services: services.size,
};
}
/**
* Clear all registrations (useful for testing)
*/
clear(): void {
this.handlers.clear();
this.configurations.clear();
this.handlerServices.clear();
}
/**
* Export registry data for debugging or persistence
*/
export(): {
handlers: Array<[string, HandlerMetadata]>;
configurations: Array<[string, HandlerConfiguration]>;
services: Array<[string, string]>;
} {
return {
handlers: Array.from(this.handlers.entries()),
configurations: Array.from(this.configurations.entries()),
services: Array.from(this.handlerServices.entries()),
};
}
/**
* Import registry data
*/
import(data: {
handlers: Array<[string, HandlerMetadata]>;
configurations: Array<[string, HandlerConfiguration]>;
services: Array<[string, string]>;
}): void {
this.clear();
for (const [name, metadata] of data.handlers) {
this.handlers.set(name, metadata);
}
for (const [name, config] of data.configurations) {
this.configurations.set(name, config);
}
for (const [handler, service] of data.services) {
this.handlerServices.set(handler, service);
}
}
}

66
libs/core/handler-registry/src/types.ts Normal file
View file

@ -0,0 +1,66 @@
/**
* Handler Registry Types
* Pure types for handler metadata and registration
*/
import type { JobHandler, ScheduledJob } from '@stock-bot/types';
/**
* Metadata for a single operation within a handler
*/
export interface OperationMetadata {
name: string;
method: string;
description?: string;
}
/**
* Metadata for a scheduled operation
*/
export interface ScheduleMetadata {
operation: string;
cronPattern: string;
priority?: number;
immediately?: boolean;
description?: string;
}
/**
* Complete metadata for a handler
*/
export interface HandlerMetadata {
name: string;
service?: string;
operations: OperationMetadata[];
schedules?: ScheduleMetadata[];
version?: string;
description?: string;
}
/**
* Handler configuration with operation implementations
*/
export interface HandlerConfiguration {
name: string;
operations: Record<string, JobHandler>;
scheduledJobs?: ScheduledJob[];
}
/**
* Registry statistics
*/
export interface RegistryStats {
handlers: number;
operations: number;
scheduledJobs: number;
services: number;
}
/**
* Handler discovery result
*/
export interface HandlerDiscoveryResult {
handler: HandlerMetadata;
constructor: any;
filePath?: string;
}

9
libs/core/handler-registry/tsconfig.json Normal file
View file

@ -0,0 +1,9 @@
{
"extends": "../../../tsconfig.lib.json",
"compilerOptions": {
"outDir": "./dist",
"rootDir": "./src"
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist", "test/**/*", "**/*.test.ts", "**/*.spec.ts"]
}

3
libs/core/handlers/package.json
View file

@ -10,10 +10,11 @@
"test": "bun test"
},
"dependencies": {
"@stock-bot/cache": "workspace:*",
"@stock-bot/config": "workspace:*",
"@stock-bot/handler-registry": "workspace:*",
"@stock-bot/logger": "workspace:*",
"@stock-bot/types": "workspace:*",
"@stock-bot/cache": "workspace:*",
"@stock-bot/utils": "workspace:*",
"mongodb": "^6.12.0"
},

145
libs/core/handlers/src/base/BaseHandler.ts
View file

@ -1,14 +1,17 @@
import type { Collection } from 'mongodb';
import { createNamespacedCache } from '@stock-bot/cache';
import { getLogger } from '@stock-bot/logger';
import type {
HandlerConfigWithSchedule,
IServiceContainer,
ExecutionContext,
IHandler
HandlerConfigWithSchedule,
HandlerMetadata,
IHandler,
IServiceContainer,
JobHandler,
ServiceTypes,
} from '@stock-bot/types';
import { fetch } from '@stock-bot/utils';
import { createNamespacedCache } from '@stock-bot/cache';
import { handlerRegistry } from '../registry/handler-registry';
// Handler registry is now injected, not imported
import { createJobHandler } from '../utils/create-job-handler';
/**
@ -38,16 +41,16 @@ export interface JobScheduleOptions {
* Provides common functionality and structure for queue/event operations
*/
export abstract class BaseHandler implements IHandler {
// Direct service properties - flattened for cleaner access
readonly logger;
readonly cache;
readonly globalCache;
readonly queue;
readonly proxy;
readonly browser;
readonly mongodb;
readonly postgres;
readonly questdb;
// Direct service properties - flattened for cleaner access with proper types
readonly logger: ServiceTypes['logger'];
readonly cache: ServiceTypes['cache'];
readonly globalCache: ServiceTypes['globalCache'];
readonly queue: ServiceTypes['queue'];
readonly proxy: ServiceTypes['proxy'];
readonly browser: ServiceTypes['browser'];
readonly mongodb: ServiceTypes['mongodb'];
readonly postgres: ServiceTypes['postgres'];
readonly questdb: ServiceTypes['questdb'];
private handlerName: string;
@ -162,7 +165,7 @@ export abstract class BaseHandler implements IHandler {
* Example: handler 'webshare' creates namespace 'webshare:api' -> keys will be 'cache:data-ingestion:webshare:api:*'
*/
protected createNamespacedCache(subNamespace: string) {
return createNamespacedCache(this.cache, `${this.handlerName}:${subNamespace}`);
return createNamespacedCache(this.cache || null, `${this.handlerName}:${subNamespace}`);
}
/**
@ -198,35 +201,35 @@ export abstract class BaseHandler implements IHandler {
return this.cache.del(`${this.handlerName}:${key}`);
}
/**
* Set global cache with key
*/
protected async globalCacheSet(key: string, value: any, ttl?: number): Promise<void> {
if (!this.globalCache) {
return;
}
return this.globalCache.set(key, value, ttl);
/**
* Set global cache with key
*/
protected async globalCacheSet(key: string, value: any, ttl?: number): Promise<void> {
if (!this.globalCache) {
return;
}
return this.globalCache.set(key, value, ttl);
}
/**
* Get global cache with key
*/
protected async globalCacheGet<T = any>(key: string): Promise<T | null> {
if (!this.globalCache) {
return null;
}
return this.globalCache.get(key);
/**
* Get global cache with key
*/
protected async globalCacheGet<T = any>(key: string): Promise<T | null> {
if (!this.globalCache) {
return null;
}
return this.globalCache.get(key);
}
/**
* Delete global cache with key
*/
protected async globalCacheDel(key: string): Promise<void> {
if (!this.globalCache) {
return;
}
return this.globalCache.del(key);
/**
* Delete global cache with key
*/
protected async globalCacheDel(key: string): Promise<void> {
if (!this.globalCache) {
return;
}
return this.globalCache.del(key);
}
/**
* Schedule operation with delay in seconds
*/
@ -238,7 +241,7 @@ export abstract class BaseHandler implements IHandler {
): Promise<void> {
return this.scheduleOperation(operation, payload, {
delay: delaySeconds * 1000,
...additionalOptions
...additionalOptions,
});
}
@ -294,27 +297,45 @@ export abstract class BaseHandler implements IHandler {
// }
/**
* Register this handler using decorator metadata
* Automatically reads @Handler, @Operation, and @QueueSchedule decorators
* Create handler configuration with job handlers
* This is used by the scanner to create the actual handler configuration
*/
register(serviceName?: string): void {
const constructor = this.constructor as any;
const handlerName = constructor.__handlerName || this.handlerName;
const operations = constructor.__operations || [];
const schedules = constructor.__schedules || [];
createHandlerConfig(): HandlerConfigWithSchedule {
const metadata = (this.constructor as typeof BaseHandler).extractMetadata();
if (!metadata) {
throw new Error('Handler metadata not found');
}
// Create operation handlers from decorator metadata
const operationHandlers: Record<string, any> = {};
for (const op of operations) {
operationHandlers[op.name] = createJobHandler(async payload => {
const operationHandlers: Record<string, JobHandler> = {};
for (const opName of metadata.operations) {
operationHandlers[opName] = createJobHandler(async (payload: any) => {
const context: ExecutionContext = {
type: 'queue',
metadata: { source: 'queue', timestamp: Date.now() },
};
return await this.execute(op.name, payload, context);
return await this.execute(opName, payload, context);
});
}
return {
name: metadata.name,
operations: operationHandlers,
scheduledJobs: metadata.scheduledJobs,
};
}
/**
* Extract handler metadata from decorators
* This returns metadata only - actual handler instances are created by the scanner
*/
static extractMetadata(): HandlerMetadata | null {
const constructor = this as any;
const handlerName = constructor.__handlerName;
if (!handlerName) return null;
const operations = constructor.__operations || [];
const schedules = constructor.__schedules || [];
// Create scheduled jobs from decorator metadata
const scheduledJobs = schedules.map((schedule: any) => {
// Find the operation name from the method name
@ -326,27 +347,15 @@ export abstract class BaseHandler implements IHandler {
priority: schedule.priority || 5,
immediately: schedule.immediately || false,
description: schedule.description || `${handlerName} ${schedule.operation}`,
payload: this.getScheduledJobPayload?.(schedule.operation),
};
});
const config: HandlerConfigWithSchedule = {
return {
name: handlerName,
operations: operationHandlers,
operations: operations.map((op: any) => op.name),
scheduledJobs,
description: constructor.__description,
};
handlerRegistry.registerWithSchedule(config, serviceName);
this.logger.info('Handler registered using decorator metadata', {
handlerName,
service: serviceName,
operations: operations.map((op: any) => ({ name: op.name, method: op.method })),
scheduledJobs: scheduledJobs.map((job: any) => ({
operation: job.operation,
cronPattern: job.cronPattern,
immediately: job.immediately,
})),
});
}
/**

3
libs/core/handlers/src/index.ts
View file

@ -2,8 +2,7 @@
export { BaseHandler, ScheduledHandler } from './base/BaseHandler';
export type { JobScheduleOptions } from './base/BaseHandler';
// Handler registry
export { handlerRegistry } from './registry/handler-registry';
// Handler registry is now in a separate package
// Utilities
export { createJobHandler } from './utils/create-job-handler';

11
libs/core/handlers/src/registry/auto-register.ts
View file

@ -6,8 +6,8 @@
import { readdirSync, statSync } from 'fs';
import { join, relative } from 'path';
import { getLogger } from '@stock-bot/logger';
import { BaseHandler } from '../base/BaseHandler';
import type { IServiceContainer } from '@stock-bot/types';
import { BaseHandler } from '../base/BaseHandler';
const logger = getLogger('handler-auto-register');
@ -123,14 +123,13 @@ export async function autoRegisterHandlers(
} else {
logger.info(`Registering handler: ${handlerName} from ${relativePath}`);
// Create instance and register
// Create instance - handlers now auto-register via decorators
const handler = new HandlerClass(services);
handler.register(serviceName);
// No need to set service ownership separately - it's done in register()
registered.push(handlerName);
logger.info(`Successfully registered handler: ${handlerName}`, { service: serviceName });
logger.info(`Successfully registered handler: ${handlerName}`, {
service: serviceName,
});
}
}
} catch (error) {

0
libs/core/handlers/src/registry/handler-registry.ts → libs/core/handlers/src/registry/handler-registry.ts.old
View file

4
libs/core/queue/package.json
View file

@ -14,9 +14,9 @@
"ioredis": "^5.3.0",
"rate-limiter-flexible": "^3.0.0",
"@stock-bot/cache": "*",
"@stock-bot/handler-registry": "*",
"@stock-bot/logger": "*",
"@stock-bot/types": "*",
"@stock-bot/handlers": "*"
"@stock-bot/types": "*"
},
"devDependencies": {
"typescript": "^5.3.0",

12
libs/core/queue/src/batch-processor.ts
View file

@ -171,7 +171,11 @@ async function processBatched<T>(
/**
* Process a batch job - loads items and creates individual jobs
*/
export async function processBatchJob(jobData: BatchJobData, queueName: string, queueManager: QueueManager): Promise<unknown> {
export async function processBatchJob(
jobData: BatchJobData,
queueName: string,
queueManager: QueueManager
): Promise<unknown> {
const queue = queueManager.getQueue(queueName);
const logger = queue.createChildLogger('batch-job', {
queueName,
@ -304,7 +308,11 @@ async function loadPayload<T>(
} | null;
}
async function cleanupPayload(key: string, queueName: string, queueManager: QueueManager): Promise<void> {
async function cleanupPayload(
key: string,
queueName: string,
queueManager: QueueManager
): Promise<void> {
const cache = queueManager.getCache(queueName);
await cache.del(key);
}

8
libs/core/queue/src/index.ts
View file

@ -8,11 +8,11 @@ export {
normalizeServiceName,
generateCachePrefix,
getFullQueueName,
parseQueueName
parseQueueName,
} from './service-utils';
// Re-export handler registry and utilities from handlers package
export { handlerRegistry, createJobHandler } from '@stock-bot/handlers';
// Re-export utilities from handlers package
export { createJobHandler } from '@stock-bot/handlers';
// Batch processing
export { processBatchJob, processItems } from './batch-processor';
@ -68,6 +68,4 @@ export type {
// Smart Queue types
SmartQueueConfig,
QueueRoute,
} from './types';

4
libs/core/queue/src/queue-manager.ts
View file

@ -76,7 +76,8 @@ export class QueueManager {
// Prepare queue configuration
const workers = mergedOptions.workers ?? this.config.defaultQueueOptions?.workers ?? 1;
const concurrency = mergedOptions.concurrency ?? this.config.defaultQueueOptions?.concurrency ?? 1;
const concurrency =
mergedOptions.concurrency ?? this.config.defaultQueueOptions?.concurrency ?? 1;
const queueConfig: QueueWorkerConfig = {
workers,
@ -180,7 +181,6 @@ export class QueueManager {
return this.queues;
}
/**
* Get statistics for all queues
*/

14
libs/core/queue/src/queue.ts
View file

@ -1,6 +1,7 @@
import { Queue as BullQueue, QueueEvents, Worker, type Job } from 'bullmq';
import { handlerRegistry } from '@stock-bot/handlers';
import type { JobData, JobOptions, ExtendedJobOptions, QueueStats, RedisConfig } from './types';
// Handler registry will be injected
import type { HandlerRegistry } from '@stock-bot/handler-registry';
import type { ExtendedJobOptions, JobData, JobOptions, QueueStats, RedisConfig } from './types';
import { getRedisConnection } from './utils';
// Logger interface for type safety
@ -17,6 +18,7 @@ export interface QueueWorkerConfig {
workers?: number;
concurrency?: number;
startWorker?: boolean;
handlerRegistry?: HandlerRegistry;
}
/**
@ -30,6 +32,7 @@ export class Queue {
private queueName: string;
private redisConfig: RedisConfig;
private readonly logger: Logger;
private readonly handlerRegistry?: HandlerRegistry;
constructor(
queueName: string,
@ -41,6 +44,7 @@ export class Queue {
this.queueName = queueName;
this.redisConfig = redisConfig;
this.logger = logger || console;
this.handlerRegistry = config.handlerRegistry;
const connection = getRedisConnection(redisConfig);
@ -338,7 +342,10 @@ export class Queue {
try {
// Look up handler in registry
const jobHandler = handlerRegistry.getOperation(handler, operation);
if (!this.handlerRegistry) {
throw new Error('Handler registry not configured for worker processing');
}
const jobHandler = this.handlerRegistry.getOperation(handler, operation);
if (!jobHandler) {
throw new Error(`No handler found for ${handler}:${operation}`);
@ -390,5 +397,4 @@ export class Queue {
getWorkerCount(): number {
return this.workers.length;
}
}

7
libs/core/queue/src/rate-limiter.ts
View file

@ -271,7 +271,12 @@ export class QueueRateLimiter {
limit,
};
} catch (error) {
this.logger.error('Failed to get rate limit status', { queueName, handler, operation, error });
this.logger.error('Failed to get rate limit status', {
queueName,
handler,
operation,
error,
});
return {
queueName,
handler,

9
libs/core/queue/src/service-cache.ts
View file

@ -1,6 +1,6 @@
import { createCache, type CacheProvider, type CacheStats } from '@stock-bot/cache';
import type { RedisConfig } from './types';
import { generateCachePrefix } from './service-utils';
import type { RedisConfig } from './types';
/**
* Service-aware cache that uses the service's Redis DB
@ -132,7 +132,11 @@ export class ServiceCache implements CacheProvider {
return this.cache.set(key, value, ttl);
}
async updateField<T = any>(key: string, updater: (current: T | null) => T, ttl?: number): Promise<T | null> {
async updateField<T = any>(
key: string,
updater: (current: T | null) => T,
ttl?: number
): Promise<T | null> {
if (this.cache.updateField) {
return this.cache.updateField(key, updater, ttl);
}
@ -162,7 +166,6 @@ export class ServiceCache implements CacheProvider {
}
}
/**
* Factory function to create service cache
*/

Some files were not shown because too many files have changed in this diff Show more