boki/stock-bot
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

trying to get simpler batcher working

Browse source
This commit is contained in:
Boki 2025-06-10 22:00:58 -04:00
parent 746a0fd949
commit 682b50d3b2
5 changed files with 82 additions and 652 deletions
Download patch file Download diff file Expand all files Collapse all files

78
docs/batch-processing-migration.md
View file

@ -1,61 +1,26 @@
# Batch Processing Migration Guide
## ✅ MIGRATION COMPLETED
The migration from the complex `BatchProcessor` class to the new functional batch processing approach has been **successfully completed**. The old `BatchProcessor` class has been removed entirely.
## Overview
The new functional batch processing approach simplifies the complex `BatchProcessor` class into simple, composable functions.
The new functional batch processing approach simplified the complex `BatchProcessor` class into simple, composable functions.
## Key Benefits
## Key Benefits Achieved
✅ **90% less code** - From 545 lines to ~200 lines
✅ **Simpler API** - Just function calls instead of class instantiation
✅ **Better performance** - Less overhead and memory usage
✅ **Same functionality** - All features preserved
✅ **Type safe** - Better TypeScript support
## Migration Examples
### Before (Complex Class-based)
```typescript
import { BatchProcessor } from '../utils/batch-processor';
const batchProcessor = new BatchProcessor(queueManager);
await batchProcessor.initialize();
const result = await batchProcessor.processItems({
items: symbols,
batchSize: 200,
totalDelayMs: 3600000,
jobNamePrefix: 'yahoo-live',
operation: 'live-data',
service: 'data-service',
provider: 'yahoo',
priority: 2,
createJobData: (symbol, index) => ({ symbol }),
useBatching: true,
removeOnComplete: 5,
removeOnFail: 3
});
```
### After (Simple Functional)
```typescript
import { processSymbols } from '../utils/batch-helpers';
const result = await processSymbols(symbols, queueManager, {
operation: 'live-data',
service: 'data-service',
provider: 'yahoo',
totalDelayMs: 3600000,
useBatching: true,
batchSize: 200,
priority: 2
});
```
✅ **No more payload conflicts** - Single consistent batch system
## Available Functions
All batch processing now uses the new functional approach:
### 1. `processItems<T>()` - Generic processing
```typescript
@ -153,22 +118,12 @@ const result = await processBatchJob(jobData, queueManager);
## Provider Migration
### Update Provider Operations
### ✅ Current Implementation
**Before:**
```typescript
'process-proxy-batch': async (payload: any) => {
const batchProcessor = new BatchProcessor(queueManager);
return await batchProcessor.processBatch(
payload,
(proxy: ProxyInfo) => ({ proxy, source: 'batch' })
);
}
```
All providers now use the new functional approach:
**After:**
```typescript
'process-proxy-batch': async (payload: any) => {
'process-batch-items': async (payload: any) => {
const { processBatchJob } = await import('../utils/batch-helpers');
return await processBatchJob(payload, queueManager);
}
@ -200,14 +155,9 @@ curl -X POST http://localhost:3002/api/test/batch-custom \
| API Complexity | High | Low | Much simpler |
| Type Safety | Medium | High | Better types |
## Backward Compatibility
## ✅ Migration Complete
The old `BatchProcessor` class is still available but deprecated. You can migrate gradually:
1. **Phase 1**: Use new functions for new features
2. **Phase 2**: Migrate existing simple use cases
3. **Phase 3**: Replace complex use cases
4. **Phase 4**: Remove old BatchProcessor
The old `BatchProcessor` class has been completely removed. All batch processing now uses the simplified functional approach.
## Common Issues & Solutions