removed configuration from index files and moved to di container

DI-CONTAINER-SIMPLIFICATION.md

@@ -0,0 +1,142 @@
+# DI Container Configuration Simplification
+
+## Overview
+
+We've simplified the dependency injection container setup across all backend services by creating a new `createServiceContainerFromConfig` function that directly accepts the AppConfig, eliminating repetitive configuration mapping code.
+
+## Before vs After
+
+### Before (30+ lines per service)
+```typescript
+// In each service's index.ts
+const awilixConfig = {
+  redis: {
+    host: config.database.dragonfly.host,
+    port: config.database.dragonfly.port,
+    db: config.database.dragonfly.db,
+  },
+  mongodb: {
+    uri: config.database.mongodb.uri,
+    database: config.database.mongodb.database,
+  },
+  postgres: {
+    host: config.database.postgres.host,
+    port: config.database.postgres.port,
+    database: config.database.postgres.database,
+    user: config.database.postgres.user,
+    password: config.database.postgres.password,
+  },
+  questdb: {
+    enabled: false,
+    host: config.database.questdb.host,
+    httpPort: config.database.questdb.httpPort,
+    pgPort: config.database.questdb.pgPort,
+    influxPort: config.database.questdb.ilpPort,
+    database: config.database.questdb.database,
+  },
+};
+
+container = createServiceContainer(awilixConfig);
+```
+
+### After (2-3 lines per service)
+```typescript
+// In each service's index.ts
+container = createServiceContainerFromConfig(config, {
+  enableQuestDB: false, // Service-specific options
+  enableMongoDB: true,
+  enablePostgres: true,
+  // ... other options
+});
+```
+
+## Benefits
+
+1. **Code Reduction**: ~30 lines reduced to 2-3 lines per service
+2. **Centralized Mapping**: Configuration structure mapping is now in one place
+3. **Service-Specific Control**: Each service can enable/disable specific components
+4. **Type Safety**: Direct use of AppConfig ensures type safety
+5. **Maintainability**: Changes to config structure only need updates in one place
+
+## Service Configurations
+
+### Data Ingestion Service
+```typescript
+container = createServiceContainerFromConfig(config, {
+  enableQuestDB: false,   // Not needed yet
+  enableMongoDB: true,    // Stores raw data
+  enablePostgres: true,   // Stores metadata
+  enableCache: true,      // For rate limiting
+  enableQueue: true,      // Job processing
+  enableBrowser: true,    // Web scraping
+  enableProxy: true,      // Proxy rotation
+});
+```
+
+### Data Pipeline Service
+```typescript
+container = createServiceContainerFromConfig(config, {
+  enableQuestDB: config.database.questdb?.enabled || false,
+  enableMongoDB: true,    // Reads raw data
+  enablePostgres: true,   // Writes processed data
+  enableCache: true,      // Query caching
+  enableQueue: true,      // Job processing
+  enableBrowser: false,   // Not needed
+  enableProxy: false,     // Not needed
+});
+```
+
+### Web API Service
+```typescript
+container = createServiceContainerFromConfig(config, {
+  enableQuestDB: false,   // Not needed
+  enableMongoDB: true,    // Reads data
+  enablePostgres: true,   // Reads data
+  enableCache: true,      // API caching
+  enableQueue: false,     // No job processing
+  enableBrowser: false,   // Not needed
+  enableProxy: false,     // Not needed
+});
+```
+
+## Implementation Details
+
+The new function in `@stock-bot/di/awilix-container.ts`:
+- Accepts the standard AppConfig from `@stock-bot/config`
+- Maps the nested config structure to what Awilix expects
+- Provides sensible defaults for all options
+- Only creates services that are enabled for the specific service
+
+## Migration Guide
+
+To migrate a service:
+
+1. Change the import:
+   ```typescript
+   // Before
+   import { createServiceContainer, ... } from '@stock-bot/di';
+   
+   // After
+   import { createServiceContainerFromConfig, ... } from '@stock-bot/di';
+   ```
+
+2. Replace the config mapping:
+   ```typescript
+   // Before
+   const awilixConfig = { /* 30+ lines of mapping */ };
+   container = createServiceContainer(awilixConfig);
+   
+   // After
+   container = createServiceContainerFromConfig(config, {
+     // Service-specific options
+   });
+   ```
+
+3. Choose which services to enable based on your service's needs
+
+## Result
+
+- **3 services updated**: data-ingestion, data-pipeline, web-api
+- **~90 lines of code removed** (30 lines × 3 services)
+- **Cleaner, more maintainable codebase**
+- **Easier to add new services** with minimal boilerplate