97 lines
No EOL
3.5 KiB
Markdown
97 lines
No EOL
3.5 KiB
Markdown
# Data-Ingestion Service Refactor Summary
|
|
|
|
## What Was Done
|
|
|
|
Successfully refactored the `data-ingestion` service to use the new connection pool pattern, completely removing dependencies on the singleton anti-pattern.
|
|
|
|
### Key Changes
|
|
|
|
1. **Service Container Setup**
|
|
- Created `database-setup.ts` with proper connection factory configuration
|
|
- Configured appropriate pool sizes for data ingestion workloads
|
|
- Added optional dynamic pool sizing for production environments
|
|
|
|
2. **Main Service Refactor** (`index.ts`)
|
|
- Removed `connectMongoDB()` and `connectPostgreSQL()` singleton calls
|
|
- Replaced with `setupServiceContainer()` initialization
|
|
- Updated shutdown handlers to dispose container properly
|
|
- Routes now have access to the service container
|
|
|
|
3. **Handler Updates**
|
|
- All handlers now accept `ServiceContainer` parameter
|
|
- QM handler operations use container-based OperationContext
|
|
- IB, Proxy, and WebShare handlers updated to accept container
|
|
- Added proper resource disposal with `ctx.dispose()`
|
|
|
|
4. **Route Refactoring**
|
|
- Created `create-routes.ts` factory function
|
|
- Routes can access container through Hono context
|
|
- Maintains backward compatibility for simple routes
|
|
|
|
5. **Migration Helper**
|
|
- Created temporary migration helper for legacy operations
|
|
- Provides `getMongoDBClient()` for IB operations still being migrated
|
|
- Includes cleanup in shutdown sequence
|
|
|
|
### Configuration Changes
|
|
|
|
- Added `@stock-bot/connection-factory` dependency
|
|
- Updated `tsconfig.json` with proper references
|
|
- Pool sizes optimized for data ingestion:
|
|
- MongoDB: 50 connections (batch imports)
|
|
- PostgreSQL: 30 connections
|
|
- Cache: 20 connections
|
|
|
|
### Benefits Achieved
|
|
|
|
1. **No More Global State**: Each service manages its own connections
|
|
2. **Better Resource Management**: Proper cleanup on shutdown
|
|
3. **Scalability**: Dynamic pool sizing for production workloads
|
|
4. **Monitoring**: Pool metrics available for observability
|
|
5. **Testing**: Easier to test with mock containers
|
|
6. **Gradual Migration**: Legacy operations still work during transition
|
|
|
|
### Next Steps
|
|
|
|
1. **Complete Operation Migration**: Update IB operations to use container
|
|
2. **Remove Migration Helper**: Once all operations are migrated
|
|
3. **Add Monitoring**: Set up dashboards for pool metrics
|
|
4. **Performance Testing**: Validate pool sizes under load
|
|
5. **Replicate Pattern**: Apply same refactor to other services
|
|
|
|
### Example Usage
|
|
|
|
```typescript
|
|
// Handler with container
|
|
export function initializeHandler(container: ServiceContainer) {
|
|
const config = {
|
|
operations: {
|
|
'my-operation': createJobHandler(async (payload) => {
|
|
// Operation uses container
|
|
const ctx = OperationContext.create('handler', 'operation', { container });
|
|
try {
|
|
// Use databases through context
|
|
await ctx.mongodb.insertOne(data);
|
|
await ctx.postgres.query('...');
|
|
return { success: true };
|
|
} finally {
|
|
await ctx.dispose(); // Clean up resources
|
|
}
|
|
})
|
|
}
|
|
};
|
|
}
|
|
```
|
|
|
|
### Migration Checklist
|
|
|
|
- [x] Remove singleton imports from index.ts
|
|
- [x] Create service container setup
|
|
- [x] Update all handlers to accept container
|
|
- [x] Create route factory with container access
|
|
- [x] Add migration helper for legacy code
|
|
- [x] Update shutdown handlers
|
|
- [x] Test build successfully
|
|
- [ ] Migrate remaining operations
|
|
- [ ] Remove migration helper
|
|
- [ ] Deploy and monitor |