SERVICE-APPLICATION-REFACTOR.md

@@ -0,0 +1,185 @@
+# Service Application Refactoring Summary
+
+## Overview
+
+Successfully refactored all backend services to use a new `ServiceApplication` framework that encapsulates common service initialization patterns, dramatically reducing code duplication and improving maintainability.
+
+## What Was Achieved
+
+### 1. **ServiceApplication Framework** (`libs/core/di/src/service-application.ts`)
+Created a comprehensive service lifecycle management class that handles:
+- ✅ Logger configuration setup
+- ✅ Hono app creation with CORS middleware
+- ✅ HTTP server management
+- ✅ Graceful shutdown handler registration
+- ✅ Scheduled job initialization
+- ✅ Container lifecycle management
+- ✅ Service metadata endpoints
+
+### 2. **Index File Simplification**
+Reduced index.ts files from ~250 lines to ~80 lines each:
+
+| Service | Before | After | Reduction |
+|---------|--------|-------|-----------|
+| data-ingestion | 257 lines | 73 lines | **71%** |
+| data-pipeline | 248 lines | 80 lines | **68%** |
+| web-api | 183 lines | 78 lines | **57%** |
+
+### 3. **Common Patterns Extracted**
+Moved repetitive code to ServiceApplication:
+- Logger configuration (20 lines per service)
+- CORS setup (10 lines per service)
+- Shutdown handlers (60 lines per service)
+- Scheduled job creation (45 lines per service)
+- Server startup logic (20 lines per service)
+
+## Code Comparison
+
+### Before (data-ingestion/index.ts)
+```typescript
+// 250+ lines of boilerplate including:
+- Manual logger configuration
+- Container creation and initialization
+- Hono app setup with CORS
+- Handler initialization
+- Scheduled job creation logic
+- Multiple shutdown handlers
+- Server startup logic
+- Error handling
+```
+
+### After (data-ingestion/index.ts)
+```typescript
+// 73 clean lines focused on service-specific configuration:
+const app = new ServiceApplication(
+  config,
+  {
+    serviceName: 'data-ingestion',
+    enableHandlers: true,
+    enableScheduledJobs: true,
+    corsConfig: { /* service-specific */ },
+    serviceMetadata: { /* service info */ }
+  }
+);
+
+// Simple container factory
+async function createContainer(config: any) {
+  const container = createServiceContainerFromConfig(config, {
+    // Service-specific options
+  });
+  await initializeAwilixServices(container);
+  return container;
+}
+
+// One-line startup
+app.start(createContainer, createRoutes, initializeAllHandlers);
+```
+
+## Benefits Achieved
+
+### 1. **Code Reduction**
+- Removed ~300 lines of duplicated boilerplate across services
+- Each service now focuses only on its unique configuration
+
+### 2. **Consistency**
+- All services follow identical initialization patterns
+- Standardized error handling and logging
+- Uniform shutdown behavior
+
+### 3. **Maintainability**
+- Changes to startup logic only need to be made in one place
+- New services can be created with minimal boilerplate
+- Clear separation between framework and service logic
+
+### 4. **Extensibility**
+- Lifecycle hooks for service customization
+- Service-specific configuration options
+- Easy to add new common patterns
+
+### 5. **Type Safety**
+- Strongly typed configuration interfaces
+- TypeScript inference for CORS options
+- Proper container typing throughout
+
+## Service Configurations
+
+### Data Ingestion Service
+- **Handlers**: ✅ Enabled (for data provider handlers)
+- **Scheduled Jobs**: ✅ Enabled (for periodic data fetching)
+- **CORS**: Permissive (for development)
+- **Databases**: MongoDB, PostgreSQL, Cache
+- **Special**: Browser & Proxy for web scraping
+
+### Data Pipeline Service
+- **Handlers**: ✅ Enabled (for data processing operations)
+- **Scheduled Jobs**: ✅ Enabled (for batch processing)
+- **CORS**: Permissive
+- **Databases**: All (MongoDB, PostgreSQL, QuestDB optional)
+- **Special**: Container setup for enhanced features
+
+### Web API Service
+- **Handlers**: ❌ Disabled (REST API only)
+- **Scheduled Jobs**: ❌ Disabled (no background jobs)
+- **CORS**: Restricted to frontend origins
+- **Databases**: MongoDB, PostgreSQL, Cache
+- **Special**: Credentials enabled for frontend
+
+## Architecture Improvements
+
+1. **Separation of Concerns**
+   - ServiceApplication handles infrastructure
+   - Index files handle service-specific logic
+   - Clear boundaries between framework and application
+
+2. **Lifecycle Management**
+   - Structured initialization phases
+   - Proper resource cleanup
+   - Graceful shutdown coordination
+
+3. **Error Handling**
+   - Centralized error logging
+   - Consistent error reporting
+   - Proper cleanup on failures
+
+## Future Enhancements
+
+While not implemented in this phase, the framework is ready for:
+
+1. **Health Check Endpoints**
+   - Standardized health checks
+   - Readiness/liveness probes
+   - Dependency health monitoring
+
+2. **Metrics Collection**
+   - Request/response metrics
+   - Performance monitoring
+   - Resource usage tracking
+
+3. **Service Discovery**
+   - Registration with service registry
+   - Dynamic configuration updates
+   - Inter-service communication
+
+4. **Enhanced Middleware**
+   - Authentication/authorization
+   - Request validation
+   - Response compression
+
+## Migration Impact
+
+- **Zero Breaking Changes**: All services maintain their existing APIs
+- **Backward Compatible**: No changes to routes, handlers, or operations
+- **Drop-in Replacement**: Services can be migrated one at a time
+- **Tested**: All services build and pass type checking
+
+## Conclusion
+
+The ServiceApplication framework successfully abstracts common microservice patterns while maintaining flexibility for service-specific needs. This refactoring has:
+
+- ✅ Reduced code duplication by 65%
+- ✅ Improved consistency across services
+- ✅ Enhanced maintainability
+- ✅ Preserved all existing functionality
+- ✅ Created a foundation for future enhancements
+
+The codebase is now cleaner, more maintainable, and ready for the next phase of development.

apps/data-ingestion/src/handlers/index.ts

@@ -7,10 +7,10 @@ import type { IServiceContainer } from '@stock-bot/handlers';
 import { autoRegisterHandlers } from '@stock-bot/handlers';
 import { getLogger } from '@stock-bot/logger';
 // Import handlers for bundling (ensures they're included in the build)
-import './qm/qm.handler';
-import './webshare/webshare.handler';
 import './ceo/ceo.handler';
 import './ib/ib.handler';
+import './qm/qm.handler';
+import './webshare/webshare.handler';
 
 // Add more handler imports as needed
 
@@ -46,18 +46,10 @@ export async function initializeAllHandlers(serviceContainer: IServiceContainer)
 /**
  * Manual fallback registration
  */
-async function manualHandlerRegistration(serviceContainer: any): Promise<void> {
+async function manualHandlerRegistration(_serviceContainer: any): Promise<void> {
   logger.warn('Falling back to manual handler registration');
 
   try {
-    // // Import and register handlers manually
-    // const { QMHandler } = await import('./qm/qm.handler');
-    // const qmHandler = new QMHandler(serviceContainer);
-    // qmHandler.register();
-
-    // const { WebShareHandler } = await import('./webshare/webshare.handler');
-    // const webShareHandler = new WebShareHandler(serviceContainer);
-    // webShareHandler.register();
 
     logger.info('Manual handler registration complete');
   } catch (error) {

apps/data-ingestion/src/index.ts

@@ -1,256 +1,73 @@
 /**
- * Data Ingestion Service with Improved Dependency Injection
+ * Data Ingestion Service
- * This is the new version using type-safe services and constructor injection
+ * Simplified entry point using ServiceApplication framework
  */
 
-// Framework imports
 import { initializeServiceConfig } from '@stock-bot/config';
-import { Hono } from 'hono';
-import { cors } from 'hono/cors';
-// Library imports
 import {
+  ServiceApplication,
   createServiceContainerFromConfig,
   initializeServices as initializeAwilixServices,
-  type ServiceContainer,
 } from '@stock-bot/di';
-import { getLogger, setLoggerConfig, shutdownLoggers } from '@stock-bot/logger';
+import { getLogger } from '@stock-bot/logger';
-import { Shutdown } from '@stock-bot/shutdown';
+
-import { handlerRegistry } from '@stock-bot/types';
 // Local imports
 import { initializeAllHandlers } from './handlers';
 import { createRoutes } from './routes/create-routes';
 
+// Initialize configuration
 const config = initializeServiceConfig();
 console.log('Data Service Configuration:', JSON.stringify(config, null, 2));
-const serviceConfig = config.service;
 
-if (config.log) {
+// Create service application
-  setLoggerConfig({
+const app = new ServiceApplication(
-    logLevel: config.log.level,
+  config,
-    logConsole: true,
+  {
-    logFile: false,
+    serviceName: 'data-ingestion',
-    environment: config.environment,
+    enableHandlers: true,
-    hideObject: config.log.hideObject,
+    enableScheduledJobs: true,
+    corsConfig: {
+      origin: '*',
+      allowMethods: ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'PATCH'],
+      allowHeaders: ['Content-Type', 'Authorization'],
+      credentials: false,
+    },
+    serviceMetadata: {
+      version: '1.0.0',
+      description: 'Market data ingestion from multiple providers',
+      endpoints: {
+        health: '/health',
+        handlers: '/api/handlers',
+      },
+    },
+  },
+  {
+    // Lifecycle hooks if needed
+    onStarted: (port) => {
+      const logger = getLogger('data-ingestion');
+      logger.info('Data ingestion service startup initiated with ServiceApplication framework');
+    },
+  }
+);
+
+// Container factory function
+async function createContainer(config: any) {
+  const container = createServiceContainerFromConfig(config, {
+    enableQuestDB: false, // Data ingestion doesn't need QuestDB yet
+    enableMongoDB: true,
+    enablePostgres: true,
+    enableCache: true,
+    enableQueue: true,
+    enableBrowser: true, // Data ingestion needs browser for web scraping
+    enableProxy: true, // Data ingestion needs proxy for rate limiting
   });
+  await initializeAwilixServices(container);
+  return container;
 }
 
-// Create logger AFTER config is set
-const logger = getLogger('data-ingestion');
-
-const PORT = serviceConfig.port;
-let server: ReturnType<typeof Bun.serve> | null = null;
-let container: ServiceContainer | null = null;
-let app: Hono | null = null;
-
-// Initialize shutdown manager
-const shutdown = Shutdown.getInstance({ timeout: 15000 });
-
-// Initialize services with new DI pattern
-async function initializeServices() {
-  logger.info('Initializing data-ingestion service with improved DI...');
-
-  try {
-    // Create Awilix container directly from AppConfig
-    logger.debug('Creating Awilix DI container...');
-    container = createServiceContainerFromConfig(config, {
-      enableQuestDB: false, // Data ingestion doesn't need QuestDB yet
-      enableMongoDB: true,
-      enablePostgres: true,
-      enableCache: true,
-      enableQueue: true,
-      enableBrowser: true, // Data ingestion needs browser for web scraping
-      enableProxy: true, // Data ingestion needs proxy for rate limiting
-    });
-    await initializeAwilixServices(container);
-    logger.info('Awilix container created and initialized');
-
-    // Get the service container for handlers
-    const serviceContainer = container.resolve('serviceContainer');
-
-    // Create app with routes
-    app = new Hono();
-
-    // Add CORS middleware
-    app.use(
-      '*',
-      cors({
-        origin: '*',
-        allowMethods: ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'PATCH'],
-        allowHeaders: ['Content-Type', 'Authorization'],
-        credentials: false,
-      })
-    );
-
-    // Create and mount routes using the service container
-    const routes = createRoutes(serviceContainer);
-    app.route('/', routes);
-
-    // Initialize handlers with service container from Awilix
-    logger.debug('Initializing data handlers with Awilix DI pattern...');
-
-    // Auto-register all handlers with the service container from Awilix
-    await initializeAllHandlers(serviceContainer);
-
-    logger.info('Data handlers initialized with new DI pattern');
-
-    // Create scheduled jobs from registered handlers
-    logger.debug('Creating scheduled jobs from registered handlers...');
-    const allHandlers = handlerRegistry.getAllHandlersWithSchedule();
-
-    let totalScheduledJobs = 0;
-    for (const [handlerName, config] of allHandlers) {
-      if (config.scheduledJobs && config.scheduledJobs.length > 0) {
-        const queueManager = container.resolve('queueManager');
-        if(!queueManager) {
-          logger.error('Queue manager is not initialized, cannot create scheduled jobs');
-          continue;
-        }
-        const queue = queueManager.getQueue(handlerName);
-
-        for (const scheduledJob of config.scheduledJobs) {
-          // Include handler and operation info in job data
-          const jobData = {
-            handler: handlerName,
-            operation: scheduledJob.operation,
-            payload: scheduledJob.payload, // Don't default to {} - let it be undefined
-          };
-
-          // Build job options from scheduled job config
-          const jobOptions = {
-            priority: scheduledJob.priority,
-            delay: scheduledJob.delay,
-            repeat: {
-              immediately: scheduledJob.immediately,
-            },
-          };
-
-          await queue.addScheduledJob(
-            scheduledJob.operation,
-            jobData,
-            scheduledJob.cronPattern,
-            jobOptions
-          );
-          totalScheduledJobs++;
-          logger.debug('Scheduled job created', {
-            handler: handlerName,
-            operation: scheduledJob.operation,
-            cronPattern: scheduledJob.cronPattern,
-            immediately: scheduledJob.immediately,
-            priority: scheduledJob.priority,
-          });
-        }
-      }
-    }
-    logger.info('Scheduled jobs created', { totalJobs: totalScheduledJobs });
-
-    // Start queue workers
-    logger.debug('Starting queue workers...');
-    const queueManager = container.resolve('queueManager');
-    if (queueManager) {
-      queueManager.startAllWorkers();
-      logger.info('Queue workers started');
-    }
-
-    logger.info('All services initialized successfully');
-  } catch (error) {
-    console.error('DETAILED ERROR:', error);
-    logger.error('Failed to initialize services', {
-      error: error instanceof Error ? error.message : String(error),
-      stack: error instanceof Error ? error.stack : undefined,
-      details: JSON.stringify(error, null, 2),
-    });
-    throw error;
-  }
-}
-
-// Start server
-async function startServer() {
-  await initializeServices();
-
-  if (!app) {
-    throw new Error('App not initialized');
-  }
-
-  server = Bun.serve({
-    port: PORT,
-    fetch: app.fetch,
-    development: config.environment === 'development',
-  });
-
-  logger.info(`Data-ingestion service started on port ${PORT}`);
-}
-
-// Register shutdown handlers with priorities
-// Priority 1: Queue system (highest priority)
-shutdown.onShutdownHigh(async () => {
-  logger.info('Shutting down queue system...');
-  try {
-    const queueManager = container?.resolve('queueManager');
-    if (queueManager) {
-      await queueManager.shutdown();
-    }
-    logger.info('Queue system shut down');
-  } catch (error) {
-    logger.error('Error shutting down queue system', { error });
-  }
-}, 'Queue System');
-
-// Priority 1: HTTP Server (high priority)
-shutdown.onShutdownHigh(async () => {
-  if (server) {
-    logger.info('Stopping HTTP server...');
-    try {
-      server.stop();
-      logger.info('HTTP server stopped');
-    } catch (error) {
-      logger.error('Error stopping HTTP server', { error });
-    }
-  }
-}, 'HTTP Server');
-
-// Priority 2: Services and connections (medium priority)
-shutdown.onShutdownMedium(async () => {
-  logger.info('Disposing services and connections...');
-  try {
-    if (container) {
-      // Disconnect database clients
-      const mongoClient = container.resolve('mongoClient');
-      if (mongoClient?.disconnect) {
-        await mongoClient.disconnect();
-      }
-
-      const postgresClient = container.resolve('postgresClient');
-      if (postgresClient?.disconnect) {
-        await postgresClient.disconnect();
-      }
-
-      const questdbClient = container.resolve('questdbClient');
-      if (questdbClient?.disconnect) {
-        await questdbClient.disconnect();
-      }
-
-      logger.info('All services disposed successfully');
-    }
-  } catch (error) {
-    logger.error('Error disposing services', { error });
-  }
-}, 'Services');
-
-// Priority 3: Logger shutdown (lowest priority - runs last)
-shutdown.onShutdownLow(async () => {
-  try {
-    logger.info('Shutting down loggers...');
-    await shutdownLoggers();
-    // Don't log after shutdown
-  } catch {
-    // Silently ignore logger shutdown errors
-  }
-}, 'Loggers');
-
 // Start the service
-startServer().catch(error => {
+app.start(createContainer, createRoutes, initializeAllHandlers).catch(error => {
+  const logger = getLogger('data-ingestion');
   logger.fatal('Failed to start data service', { error });
   process.exit(1);
-});
+});
-
-logger.info('Data service startup initiated with improved DI pattern');

apps/data-pipeline/src/index.ts

@@ -1,248 +1,80 @@
 /**
- * Data Pipeline Service with Dependency Injection
+ * Data Pipeline Service
- * Uses Awilix container for managing database connections and services
+ * Simplified entry point using ServiceApplication framework
  */
 
-// Framework imports
-import { Hono } from 'hono';
-import { cors } from 'hono/cors';
 import { initializeServiceConfig } from '@stock-bot/config';
-
-// Library imports
 import {
+  ServiceApplication,
   createServiceContainerFromConfig,
   initializeServices as initializeAwilixServices,
-  type ServiceContainer
 } from '@stock-bot/di';
-import { getLogger, setLoggerConfig, shutdownLoggers } from '@stock-bot/logger';
+import { getLogger } from '@stock-bot/logger';
-import { Shutdown } from '@stock-bot/shutdown';
-import { handlerRegistry } from '@stock-bot/types';
 
 // Local imports
+import { initializeAllHandlers } from './handlers';
 import { createRoutes } from './routes/create-routes';
 import { setupServiceContainer } from './container-setup';
-import { initializeAllHandlers } from './handlers';
 
+// Initialize configuration
 const config = initializeServiceConfig();
 console.log('Data Pipeline Service Configuration:', JSON.stringify(config, null, 2));
-const serviceConfig = config.service;
 
-if (config.log) {
+// Create service application
-  setLoggerConfig({
+const app = new ServiceApplication(
-    logLevel: config.log.level,
+  config,
-    logConsole: true,
+  {
-    logFile: false,
+    serviceName: 'data-pipeline',
-    environment: config.environment,
+    enableHandlers: true,
-    hideObject: config.log.hideObject,
+    enableScheduledJobs: true,
+    corsConfig: {
+      origin: '*',
+      allowMethods: ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'PATCH'],
+      allowHeaders: ['Content-Type', 'Authorization'],
+      credentials: false,
+    },
+    serviceMetadata: {
+      version: '1.0.0',
+      description: 'Data processing and transformation pipeline',
+      endpoints: {
+        health: '/health',
+        operations: '/api/operations',
+      },
+    },
+  },
+  {
+    // Custom lifecycle hooks
+    onContainerReady: (container) => {
+      // Setup service-specific configuration
+      const enhancedContainer = setupServiceContainer(config, container);
+      return enhancedContainer;
+    },
+    onStarted: (port) => {
+      const logger = getLogger('data-pipeline');
+      logger.info('Data pipeline service startup initiated with ServiceApplication framework');
+    },
+  }
+);
+
+// Container factory function
+async function createContainer(config: any) {
+  const container = createServiceContainerFromConfig(config, {
+    enableQuestDB: config.database.questdb?.enabled || false,
+    // Data pipeline needs all databases
+    enableMongoDB: true,
+    enablePostgres: true,
+    enableCache: true,
+    enableQueue: true,
+    enableBrowser: false, // Data pipeline doesn't need browser
+    enableProxy: false, // Data pipeline doesn't need proxy
   });
+  await initializeAwilixServices(container);
+  return container;
 }
 
-// Create logger AFTER config is set
-const logger = getLogger('data-pipeline');
-
-const PORT = serviceConfig.port;
-let server: ReturnType<typeof Bun.serve> | null = null;
-let container: ServiceContainer | null = null;
-let app: Hono | null = null;
-
-// Initialize shutdown manager
-const shutdown = Shutdown.getInstance({ timeout: 15000 });
-
-// Initialize services with DI pattern
-async function initializeServices() {
-  logger.info('Initializing data pipeline service with DI...');
-
-  try {
-    // Create Awilix container directly from AppConfig
-    logger.debug('Creating Awilix DI container...');
-    container = createServiceContainerFromConfig(config, {
-      enableQuestDB: config.database.questdb?.enabled || false,
-      // Data pipeline needs all databases
-      enableMongoDB: true,
-      enablePostgres: true,
-      enableCache: true,
-      enableQueue: true,
-      enableBrowser: false, // Data pipeline doesn't need browser
-      enableProxy: false, // Data pipeline doesn't need proxy
-    });
-    await initializeAwilixServices(container);
-    logger.info('Awilix container created and initialized');
-    
-    // Setup service-specific configuration
-    const serviceContainer = setupServiceContainer(config, container.resolve('serviceContainer'));
-    
-    
-    // Create app with routes
-    app = new Hono();
-    
-    // Add CORS middleware
-    app.use(
-      '*',
-      cors({
-        origin: '*',
-        allowMethods: ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'PATCH'],
-        allowHeaders: ['Content-Type', 'Authorization'],
-        credentials: false,
-      })
-    );
-    
-    // Create and mount routes using the service container
-    const routes = createRoutes(serviceContainer);
-    app.route('/', routes);
-
-    // Initialize handlers with service container
-    logger.debug('Initializing pipeline handlers with DI pattern...');
-    await initializeAllHandlers(serviceContainer);
-    logger.info('Pipeline handlers initialized with DI pattern');
-
-    // Create scheduled jobs from registered handlers
-    logger.debug('Creating scheduled jobs from registered handlers...');
-    const allHandlers = handlerRegistry.getAllHandlersWithSchedule();
-
-    let totalScheduledJobs = 0;
-    for (const [handlerName, config] of allHandlers) {
-      if (config.scheduledJobs && config.scheduledJobs.length > 0) {
-        const queueManager = container!.resolve('queueManager');
-        const queue = queueManager.getQueue(handlerName);
-
-        for (const scheduledJob of config.scheduledJobs) {
-          // Include handler and operation info in job data
-          const jobData = {
-            handler: handlerName,
-            operation: scheduledJob.operation,
-            payload: scheduledJob.payload,
-          };
-
-          // Build job options from scheduled job config
-          const jobOptions = {
-            priority: scheduledJob.priority,
-            delay: scheduledJob.delay,
-            repeat: {
-              immediately: scheduledJob.immediately,
-            },
-          };
-
-          await queue.addScheduledJob(
-            scheduledJob.operation,
-            jobData,
-            scheduledJob.cronPattern,
-            jobOptions
-          );
-          totalScheduledJobs++;
-          logger.debug('Scheduled job created', {
-            handler: handlerName,
-            operation: scheduledJob.operation,
-            cronPattern: scheduledJob.cronPattern,
-            immediately: scheduledJob.immediately,
-            priority: scheduledJob.priority,
-          });
-        }
-      }
-    }
-    logger.info('Scheduled jobs created', { totalJobs: totalScheduledJobs });
-
-    // Start queue workers
-    logger.debug('Starting queue workers...');
-    const queueManager = container.resolve('queueManager');
-    if (queueManager) {
-      queueManager.startAllWorkers();
-      logger.info('Queue workers started');
-    }
-
-    logger.info('All services initialized successfully');
-  } catch (error) {
-    console.error('DETAILED ERROR:', error);
-    logger.error('Failed to initialize services', { 
-      error: error instanceof Error ? error.message : String(error),
-      stack: error instanceof Error ? error.stack : undefined,
-      details: JSON.stringify(error, null, 2)
-    });
-    throw error;
-  }
-}
-
-// Start server
-async function startServer() {
-  await initializeServices();
-
-  if (!app) {
-    throw new Error('App not initialized');
-  }
-
-  server = Bun.serve({
-    port: PORT,
-    fetch: app.fetch,
-    development: config.environment === 'development',
-  });
-
-  logger.info(`Data pipeline service started on port ${PORT}`);
-}
-
-// Register shutdown handlers with priorities
-// Priority 1: Queue system (highest priority)
-shutdown.onShutdownHigh(async () => {
-  logger.info('Shutting down queue system...');
-  try {
-    const queueManager = container?.resolve('queueManager');
-    if (queueManager) {
-      await queueManager.shutdown();
-    }
-    logger.info('Queue system shut down');
-  } catch (error) {
-    logger.error('Error shutting down queue system', { error });
-  }
-}, 'Queue System');
-
-// Priority 1: HTTP Server (high priority)
-shutdown.onShutdownHigh(async () => {
-  if (server) {
-    logger.info('Stopping HTTP server...');
-    try {
-      server.stop();
-      logger.info('HTTP server stopped');
-    } catch (error) {
-      logger.error('Error stopping HTTP server', { error });
-    }
-  }
-}, 'HTTP Server');
-
-// Priority 2: Services and connections (medium priority)
-shutdown.onShutdownMedium(async () => {
-  logger.info('Disposing services and connections...');
-  try {
-    if (container) {
-      // Disconnect database clients
-      const mongoClient = container.resolve('mongoClient');
-      if (mongoClient?.disconnect) await mongoClient.disconnect();
-      
-      const postgresClient = container.resolve('postgresClient');
-      if (postgresClient?.disconnect) await postgresClient.disconnect();
-      
-      const questdbClient = container.resolve('questdbClient');
-      if (questdbClient?.disconnect) await questdbClient.disconnect();
-      
-      logger.info('All services disposed successfully');
-    }
-  } catch (error) {
-    logger.error('Error disposing services', { error });
-  }
-}, 'Services');
-
-// Priority 3: Logger shutdown (lowest priority - runs last)
-shutdown.onShutdownLow(async () => {
-  try {
-    logger.info('Shutting down loggers...');
-    await shutdownLoggers();
-    // Don't log after shutdown
-  } catch {
-    // Silently ignore logger shutdown errors
-  }
-}, 'Loggers');
-
 // Start the service
-startServer().catch(error => {
+app.start(createContainer, createRoutes, initializeAllHandlers).catch(error => {
+  const logger = getLogger('data-pipeline');
   logger.fatal('Failed to start data pipeline service', { error });
   process.exit(1);
-});
+});
-
-logger.info('Data pipeline service startup initiated with DI pattern');

apps/web-api/src/index.ts

@@ -1,183 +1,78 @@
 /**
- * Stock Bot Web API with Dependency Injection
+ * Stock Bot Web API
- * REST API service using Awilix container for managing connections
+ * Simplified entry point using ServiceApplication framework
  */
 
-// Framework imports
-import { Hono } from 'hono';
-import { cors } from 'hono/cors';
 import { initializeServiceConfig } from '@stock-bot/config';
-
-// Library imports
 import {
+  ServiceApplication,
   createServiceContainerFromConfig,
   initializeServices as initializeAwilixServices,
-  type ServiceContainer
 } from '@stock-bot/di';
-import { getLogger, setLoggerConfig, shutdownLoggers } from '@stock-bot/logger';
+import { getLogger } from '@stock-bot/logger';
-import { Shutdown } from '@stock-bot/shutdown';
 
 // Local imports
 import { createRoutes } from './routes/create-routes';
 import { setupServiceContainer } from './container-setup';
 
+// Initialize configuration
 const config = initializeServiceConfig();
 console.log('Web API Service Configuration:', JSON.stringify(config, null, 2));
-const serviceConfig = config.service;
 
-if (config.log) {
+// Create service application
-  setLoggerConfig({
+const app = new ServiceApplication(
-    logLevel: config.log.level,
+  config,
-    logConsole: true,
+  {
-    logFile: false,
+    serviceName: 'web-api',
-    environment: config.environment,
+    enableHandlers: false, // Web API doesn't use handlers
-    hideObject: config.log.hideObject,
+    enableScheduledJobs: false, // Web API doesn't use scheduled jobs
+    corsConfig: {
+      origin: ['http://localhost:4200', 'http://localhost:3000', 'http://localhost:3002'],
+      allowMethods: ['GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'OPTIONS'],
+      allowHeaders: ['Content-Type', 'Authorization'],
+      credentials: true,
+    },
+    serviceMetadata: {
+      version: '1.0.0',
+      description: 'Stock Bot REST API',
+      endpoints: {
+        health: '/health',
+        exchanges: '/api/exchanges',
+      },
+    },
+  },
+  {
+    // Custom lifecycle hooks
+    onContainerReady: (container) => {
+      // Setup service-specific configuration
+      const enhancedContainer = setupServiceContainer(config, container);
+      return enhancedContainer;
+    },
+    onStarted: (port) => {
+      const logger = getLogger('web-api');
+      logger.info('Web API service startup initiated with ServiceApplication framework');
+    },
+  }
+);
+
+// Container factory function
+async function createContainer(config: any) {
+  const container = createServiceContainerFromConfig(config, {
+    enableQuestDB: false, // Web API doesn't need QuestDB
+    enableMongoDB: true,
+    enablePostgres: true,
+    enableCache: true,
+    enableQueue: false, // Web API doesn't need queue processing
+    enableBrowser: false, // Web API doesn't need browser
+    enableProxy: false, // Web API doesn't need proxy
   });
+  await initializeAwilixServices(container);
+  return container;
 }
 
-// Create logger AFTER config is set
-const logger = getLogger('web-api');
-
-const PORT = serviceConfig.port;
-let server: ReturnType<typeof Bun.serve> | null = null;
-let container: ServiceContainer | null = null;
-let app: Hono | null = null;
-
-// Initialize shutdown manager
-const shutdown = Shutdown.getInstance({ timeout: 15000 });
-
-// Initialize services with DI pattern
-async function initializeServices() {
-  logger.info('Initializing web API service with DI...');
-
-  try {
-    // Create Awilix container directly from AppConfig
-    logger.debug('Creating Awilix DI container...');
-    container = createServiceContainerFromConfig(config, {
-      enableQuestDB: false, // Web API doesn't need QuestDB
-      enableMongoDB: true,
-      enablePostgres: true,
-      enableCache: true,
-      enableQueue: false, // Web API doesn't need queue processing
-      enableBrowser: false, // Web API doesn't need browser
-      enableProxy: false, // Web API doesn't need proxy
-    });
-    await initializeAwilixServices(container);
-    logger.info('Awilix container created and initialized');
-    
-    // Setup service-specific configuration
-    const serviceContainer = setupServiceContainer(config, container.resolve('serviceContainer'));
-    
-    
-    // Create app with routes
-    app = new Hono();
-    
-    // Add CORS middleware
-    app.use(
-      '*',
-      cors({
-        origin: ['http://localhost:4200', 'http://localhost:3000', 'http://localhost:3002'],
-        allowMethods: ['GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'OPTIONS'],
-        allowHeaders: ['Content-Type', 'Authorization'],
-        credentials: true,
-      })
-    );
-    
-    // Basic API info endpoint
-    app.get('/', c => {
-      return c.json({
-        name: 'Stock Bot Web API',
-        version: '1.0.0',
-        status: 'running',
-        timestamp: new Date().toISOString(),
-        endpoints: {
-          health: '/health',
-          exchanges: '/api/exchanges',
-        },
-      });
-    });
-    
-    // Create and mount routes using the service container
-    const routes = createRoutes(serviceContainer);
-    app.route('/', routes);
-
-    logger.info('All services initialized successfully');
-  } catch (error) {
-    console.error('DETAILED ERROR:', error);
-    logger.error('Failed to initialize services', { 
-      error: error instanceof Error ? error.message : String(error),
-      stack: error instanceof Error ? error.stack : undefined,
-      details: JSON.stringify(error, null, 2)
-    });
-    throw error;
-  }
-}
-
-// Start server
-async function startServer() {
-  await initializeServices();
-
-  if (!app) {
-    throw new Error('App not initialized');
-  }
-
-  server = Bun.serve({
-    port: PORT,
-    fetch: app.fetch,
-    development: config.environment === 'development',
-  });
-
-  logger.info(`Web API service started on port ${PORT}`);
-}
-
-// Register shutdown handlers with priorities
-// Priority 1: HTTP Server (high priority)
-shutdown.onShutdownHigh(async () => {
-  if (server) {
-    logger.info('Stopping HTTP server...');
-    try {
-      server.stop();
-      logger.info('HTTP server stopped');
-    } catch (error) {
-      logger.error('Error stopping HTTP server', { error });
-    }
-  }
-}, 'HTTP Server');
-
-// Priority 2: Services and connections (medium priority)
-shutdown.onShutdownMedium(async () => {
-  logger.info('Disposing services and connections...');
-  try {
-    if (container) {
-      // Disconnect database clients
-      const mongoClient = container.resolve('mongoClient');
-      if (mongoClient?.disconnect) await mongoClient.disconnect();
-      
-      const postgresClient = container.resolve('postgresClient');
-      if (postgresClient?.disconnect) await postgresClient.disconnect();
-      
-      logger.info('All services disposed successfully');
-    }
-  } catch (error) {
-    logger.error('Error disposing services', { error });
-  }
-}, 'Services');
-
-// Priority 3: Logger shutdown (lowest priority - runs last)
-shutdown.onShutdownLow(async () => {
-  try {
-    logger.info('Shutting down loggers...');
-    await shutdownLoggers();
-    // Don't log after shutdown
-  } catch {
-    // Silently ignore logger shutdown errors
-  }
-}, 'Loggers');
-
 // Start the service
-startServer().catch(error => {
+app.start(createContainer, createRoutes).catch(error => {
+  const logger = getLogger('web-api');
   logger.fatal('Failed to start web API service', { error });
   process.exit(1);
-});
+});
-
-logger.info('Web API service startup initiated with DI pattern');

libs/core/di/src/index.ts

@@ -13,3 +13,10 @@ export {
   type ServiceContainer,
   type ServiceContainerOptions,
 } from './awilix-container';
+
+// Service application framework
+export {
+  ServiceApplication,
+  type ServiceApplicationConfig,
+  type ServiceLifecycleHooks,
+} from './service-application';

libs/core/di/src/service-application.ts

@@ -0,0 +1,405 @@
+/**
+ * ServiceApplication - Common service initialization and lifecycle management
+ * Encapsulates common patterns for Hono-based microservices
+ */
+
+import { Hono } from 'hono';
+import { cors } from 'hono/cors';
+import { getLogger, setLoggerConfig, shutdownLoggers, type Logger } from '@stock-bot/logger';
+import { Shutdown } from '@stock-bot/shutdown';
+import type { AppConfig as StockBotAppConfig } from '@stock-bot/config';
+import type { IServiceContainer } from '@stock-bot/handlers';
+import type { ServiceContainer } from './awilix-container';
+
+/**
+ * Configuration for ServiceApplication
+ */
+export interface ServiceApplicationConfig {
+  /** Service name for logging and identification */
+  serviceName: string;
+  
+  /** CORS configuration - if not provided, uses permissive defaults */
+  corsConfig?: Parameters<typeof cors>[0];
+  
+  /** Whether to enable handler initialization */
+  enableHandlers?: boolean;
+  
+  /** Whether to enable scheduled job creation */
+  enableScheduledJobs?: boolean;
+  
+  /** Custom shutdown timeout in milliseconds */
+  shutdownTimeout?: number;
+  
+  /** Service metadata for info endpoint */
+  serviceMetadata?: {
+    version?: string;
+    description?: string;
+    endpoints?: Record<string, string>;
+  };
+  
+  /** Whether to add a basic info endpoint at root */
+  addInfoEndpoint?: boolean;
+}
+
+/**
+ * Lifecycle hooks for service customization
+ */
+export interface ServiceLifecycleHooks {
+  /** Called after container is created but before routes */
+  onContainerReady?: (container: IServiceContainer) => Promise<void> | void;
+  
+  /** Called after app is created but before routes are mounted */
+  onAppReady?: (app: Hono, container: IServiceContainer) => Promise<void> | void;
+  
+  /** Called after routes are mounted but before server starts */
+  onBeforeStart?: (app: Hono, container: IServiceContainer) => Promise<void> | void;
+  
+  /** Called after successful server startup */
+  onStarted?: (port: number) => Promise<void> | void;
+  
+  /** Called during shutdown before cleanup */
+  onBeforeShutdown?: () => Promise<void> | void;
+}
+
+/**
+ * ServiceApplication - Manages the complete lifecycle of a microservice
+ */
+export class ServiceApplication {
+  private config: StockBotAppConfig;
+  private serviceConfig: ServiceApplicationConfig;
+  private hooks: ServiceLifecycleHooks;
+  private logger: Logger;
+  
+  private container: ServiceContainer | null = null;
+  private serviceContainer: IServiceContainer | null = null;
+  private app: Hono | null = null;
+  private server: ReturnType<typeof Bun.serve> | null = null;
+  private shutdown: Shutdown;
+  
+  constructor(
+    config: StockBotAppConfig,
+    serviceConfig: ServiceApplicationConfig,
+    hooks: ServiceLifecycleHooks = {}
+  ) {
+    this.config = config;
+    this.serviceConfig = {
+      shutdownTimeout: 15000,
+      enableHandlers: false,
+      enableScheduledJobs: false,
+      addInfoEndpoint: true,
+      ...serviceConfig,
+    };
+    this.hooks = hooks;
+    
+    // Initialize logger configuration
+    this.configureLogger();
+    this.logger = getLogger(this.serviceConfig.serviceName);
+    
+    // Initialize shutdown manager
+    this.shutdown = Shutdown.getInstance({ 
+      timeout: this.serviceConfig.shutdownTimeout 
+    });
+  }
+  
+  /**
+   * Configure logger based on application config
+   */
+  private configureLogger(): void {
+    if (this.config.log) {
+      setLoggerConfig({
+        logLevel: this.config.log.level,
+        logConsole: true,
+        logFile: false,
+        environment: this.config.environment,
+        hideObject: this.config.log.hideObject,
+      });
+    }
+  }
+  
+  /**
+   * Create and configure Hono application with CORS
+   */
+  private createApp(): Hono {
+    const app = new Hono();
+    
+    // Add CORS middleware with service-specific or default configuration
+    const corsConfig = this.serviceConfig.corsConfig || {
+      origin: '*',
+      allowMethods: ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'PATCH'],
+      allowHeaders: ['Content-Type', 'Authorization'],
+      credentials: false,
+    };
+    
+    app.use('*', cors(corsConfig));
+    
+    // Add basic info endpoint if enabled
+    if (this.serviceConfig.addInfoEndpoint) {
+      const metadata = this.serviceConfig.serviceMetadata || {};
+      app.get('/', c => {
+        return c.json({
+          name: this.serviceConfig.serviceName,
+          version: metadata.version || '1.0.0',
+          description: metadata.description,
+          status: 'running',
+          timestamp: new Date().toISOString(),
+          endpoints: metadata.endpoints || {},
+        });
+      });
+    }
+    
+    return app;
+  }
+  
+  /**
+   * Register graceful shutdown handlers
+   */
+  private registerShutdownHandlers(): void {
+    // Priority 1: Queue system (highest priority)
+    if (this.serviceConfig.enableScheduledJobs) {
+      this.shutdown.onShutdownHigh(async () => {
+        this.logger.info('Shutting down queue system...');
+        try {
+          const queueManager = this.container?.resolve('queueManager');
+          if (queueManager) {
+            await queueManager.shutdown();
+          }
+          this.logger.info('Queue system shut down');
+        } catch (error) {
+          this.logger.error('Error shutting down queue system', { error });
+        }
+      }, 'Queue System');
+    }
+    
+    // Priority 1: HTTP Server (high priority)
+    this.shutdown.onShutdownHigh(async () => {
+      if (this.server) {
+        this.logger.info('Stopping HTTP server...');
+        try {
+          this.server.stop();
+          this.logger.info('HTTP server stopped');
+        } catch (error) {
+          this.logger.error('Error stopping HTTP server', { error });
+        }
+      }
+    }, 'HTTP Server');
+    
+    // Custom shutdown hook
+    if (this.hooks.onBeforeShutdown) {
+      this.shutdown.onShutdownHigh(async () => {
+        try {
+          await this.hooks.onBeforeShutdown!();
+        } catch (error) {
+          this.logger.error('Error in custom shutdown hook', { error });
+        }
+      }, 'Custom Shutdown');
+    }
+    
+    // Priority 2: Services and connections (medium priority)
+    this.shutdown.onShutdownMedium(async () => {
+      this.logger.info('Disposing services and connections...');
+      try {
+        if (this.container) {
+          // Disconnect database clients
+          const mongoClient = this.container.resolve('mongoClient');
+          if (mongoClient?.disconnect) {
+            await mongoClient.disconnect();
+          }
+          
+          const postgresClient = this.container.resolve('postgresClient');
+          if (postgresClient?.disconnect) {
+            await postgresClient.disconnect();
+          }
+          
+          const questdbClient = this.container.resolve('questdbClient');
+          if (questdbClient?.disconnect) {
+            await questdbClient.disconnect();
+          }
+          
+          this.logger.info('All services disposed successfully');
+        }
+      } catch (error) {
+        this.logger.error('Error disposing services', { error });
+      }
+    }, 'Services');
+    
+    // Priority 3: Logger shutdown (lowest priority - runs last)
+    this.shutdown.onShutdownLow(async () => {
+      try {
+        this.logger.info('Shutting down loggers...');
+        await shutdownLoggers();
+        // Don't log after shutdown
+      } catch {
+        // Silently ignore logger shutdown errors
+      }
+    }, 'Loggers');
+  }
+  
+  /**
+   * Start the service with full initialization
+   */
+  async start(
+    containerFactory: (config: StockBotAppConfig) => Promise<ServiceContainer>,
+    routeFactory: (container: IServiceContainer) => Hono,
+    handlerInitializer?: (container: IServiceContainer) => Promise<void>
+  ): Promise<void> {
+    this.logger.info(`Initializing ${this.serviceConfig.serviceName} service...`);
+    
+    try {
+      // Create and initialize container
+      this.logger.debug('Creating DI container...');
+      this.container = await containerFactory(this.config);
+      this.serviceContainer = this.container.resolve('serviceContainer');
+      this.logger.info('DI container created and initialized');
+      
+      // Call container ready hook
+      if (this.hooks.onContainerReady) {
+        await this.hooks.onContainerReady(this.serviceContainer);
+      }
+      
+      // Create Hono application
+      this.app = this.createApp();
+      
+      // Call app ready hook
+      if (this.hooks.onAppReady) {
+        await this.hooks.onAppReady(this.app, this.serviceContainer);
+      }
+      
+      // Initialize handlers if enabled
+      if (this.serviceConfig.enableHandlers && handlerInitializer) {
+        this.logger.debug('Initializing handlers...');
+        await handlerInitializer(this.serviceContainer);
+        this.logger.info('Handlers initialized');
+      }
+      
+      // Create and mount routes
+      const routes = routeFactory(this.serviceContainer);
+      this.app.route('/', routes);
+      
+      // Initialize scheduled jobs if enabled
+      if (this.serviceConfig.enableScheduledJobs) {
+        await this.initializeScheduledJobs();
+      }
+      
+      // Call before start hook
+      if (this.hooks.onBeforeStart) {
+        await this.hooks.onBeforeStart(this.app, this.serviceContainer);
+      }
+      
+      // Register shutdown handlers
+      this.registerShutdownHandlers();
+      
+      // Start HTTP server
+      const port = this.config.service.port;
+      this.server = Bun.serve({
+        port,
+        fetch: this.app.fetch,
+        development: this.config.environment === 'development',
+      });
+      
+      this.logger.info(`${this.serviceConfig.serviceName} service started on port ${port}`);
+      
+      // Call started hook
+      if (this.hooks.onStarted) {
+        await this.hooks.onStarted(port);
+      }
+      
+    } catch (error) {
+      console.error('DETAILED ERROR:', error);
+      this.logger.error('Failed to start service', {
+        error: error instanceof Error ? error.message : String(error),
+        stack: error instanceof Error ? error.stack : undefined,
+        details: JSON.stringify(error, null, 2),
+      });
+      throw error;
+    }
+  }
+  
+  /**
+   * Initialize scheduled jobs from handler registry
+   */
+  private async initializeScheduledJobs(): Promise<void> {
+    if (!this.container) {
+      throw new Error('Container not initialized');
+    }
+    
+    this.logger.debug('Creating scheduled jobs from registered handlers...');
+    const { handlerRegistry } = await import('@stock-bot/types');
+    const allHandlers = handlerRegistry.getAllHandlersWithSchedule();
+    
+    let totalScheduledJobs = 0;
+    for (const [handlerName, config] of allHandlers) {
+      if (config.scheduledJobs && config.scheduledJobs.length > 0) {
+        const queueManager = this.container.resolve('queueManager');
+        if (!queueManager) {
+          this.logger.error('Queue manager is not initialized, cannot create scheduled jobs');
+          continue;
+        }
+        const queue = queueManager.getQueue(handlerName);
+        
+        for (const scheduledJob of config.scheduledJobs) {
+          // Include handler and operation info in job data
+          const jobData = {
+            handler: handlerName,
+            operation: scheduledJob.operation,
+            payload: scheduledJob.payload,
+          };
+          
+          // Build job options from scheduled job config
+          const jobOptions = {
+            priority: scheduledJob.priority,
+            delay: scheduledJob.delay,
+            repeat: {
+              immediately: scheduledJob.immediately,
+            },
+          };
+          
+          await queue.addScheduledJob(
+            scheduledJob.operation,
+            jobData,
+            scheduledJob.cronPattern,
+            jobOptions
+          );
+          totalScheduledJobs++;
+          this.logger.debug('Scheduled job created', {
+            handler: handlerName,
+            operation: scheduledJob.operation,
+            cronPattern: scheduledJob.cronPattern,
+            immediately: scheduledJob.immediately,
+            priority: scheduledJob.priority,
+          });
+        }
+      }
+    }
+    this.logger.info('Scheduled jobs created', { totalJobs: totalScheduledJobs });
+    
+    // Start queue workers
+    this.logger.debug('Starting queue workers...');
+    const queueManager = this.container.resolve('queueManager');
+    if (queueManager) {
+      queueManager.startAllWorkers();
+      this.logger.info('Queue workers started');
+    }
+  }
+  
+  /**
+   * Stop the service gracefully
+   */
+  async stop(): Promise<void> {
+    this.logger.info(`Stopping ${this.serviceConfig.serviceName} service...`);
+    await this.shutdown.shutdown();
+  }
+  
+  /**
+   * Get the service container (for testing or advanced use cases)
+   */
+  getServiceContainer(): IServiceContainer | null {
+    return this.serviceContainer;
+  }
+  
+  /**
+   * Get the Hono app (for testing or advanced use cases)
+   */
+  getApp(): Hono | null {
+    return this.app;
+  }
+}