simplified things
This commit is contained in:
parent
cdc2f44e86
commit
5c269453f2
8 changed files with 4 additions and 869 deletions
|
|
@ -1,176 +0,0 @@
|
|||
# Batch Processing Migration Guide
|
||||
|
||||
## ✅ MIGRATION COMPLETED
|
||||
|
||||
The migration from the complex `BatchProcessor` class to the new functional batch processing approach has been **successfully completed**. The old `BatchProcessor` class has been removed entirely.
|
||||
|
||||
## Overview
|
||||
|
||||
The new functional batch processing approach simplified the complex `BatchProcessor` class into simple, composable functions.
|
||||
|
||||
## Key Benefits Achieved
|
||||
|
||||
✅ **90% less code** - From 545 lines to ~200 lines
|
||||
✅ **Simpler API** - Just function calls instead of class instantiation
|
||||
✅ **Better performance** - Less overhead and memory usage
|
||||
✅ **Same functionality** - All features preserved
|
||||
✅ **Type safe** - Better TypeScript support
|
||||
✅ **No more payload conflicts** - Single consistent batch system
|
||||
|
||||
## Available Functions
|
||||
|
||||
All batch processing now uses the new functional approach:
|
||||
|
||||
### 1. `processItems<T>()` - Generic processing
|
||||
|
||||
```typescript
|
||||
import { processItems } from '../utils/batch-helpers';
|
||||
|
||||
const result = await processItems(
|
||||
items,
|
||||
(item, index) => ({ /* transform item */ }),
|
||||
queueManager,
|
||||
{
|
||||
totalDelayMs: 60000,
|
||||
useBatching: false,
|
||||
batchSize: 100,
|
||||
priority: 1
|
||||
}
|
||||
);
|
||||
```
|
||||
|
||||
### 2. `processSymbols()` - Stock symbol processing
|
||||
|
||||
```typescript
|
||||
import { processSymbols } from '../utils/batch-helpers';
|
||||
|
||||
const result = await processSymbols(['AAPL', 'GOOGL'], queueManager, {
|
||||
operation: 'live-data',
|
||||
service: 'market-data',
|
||||
provider: 'yahoo',
|
||||
totalDelayMs: 300000,
|
||||
useBatching: false,
|
||||
priority: 1,
|
||||
service: 'market-data',
|
||||
provider: 'yahoo',
|
||||
operation: 'live-data'
|
||||
});
|
||||
```
|
||||
|
||||
### 3. `processBatchJob()` - Worker batch handler
|
||||
|
||||
```typescript
|
||||
import { processBatchJob } from '../utils/batch-helpers';
|
||||
|
||||
// In your worker job handler
|
||||
const result = await processBatchJob(jobData, queueManager);
|
||||
```
|
||||
|
||||
## Configuration Mapping
|
||||
|
||||
| Old BatchConfig | New ProcessOptions | Description |
|
||||
|----------------|-------------------|-------------|
|
||||
| `items` | First parameter | Items to process |
|
||||
| `createJobData` | Second parameter | Transform function |
|
||||
| `queueManager` | Third parameter | Queue instance |
|
||||
| `totalDelayMs` | `totalDelayMs` | Total processing time |
|
||||
| `batchSize` | `batchSize` | Items per batch |
|
||||
| `useBatching` | `useBatching` | Batch vs direct mode |
|
||||
| `priority` | `priority` | Job priority |
|
||||
| `removeOnComplete` | `removeOnComplete` | Job cleanup |
|
||||
| `removeOnFail` | `removeOnFail` | Failed job cleanup |
|
||||
| `payloadTtlHours` | `ttl` | Cache TTL in seconds |
|
||||
|
||||
## Return Value Changes
|
||||
|
||||
### Before
|
||||
```typescript
|
||||
{
|
||||
totalItems: number,
|
||||
jobsCreated: number,
|
||||
mode: 'direct' | 'batch',
|
||||
optimized?: boolean,
|
||||
batchJobsCreated?: number,
|
||||
// ... other complex fields
|
||||
}
|
||||
```
|
||||
|
||||
### After
|
||||
```typescript
|
||||
{
|
||||
jobsCreated: number,
|
||||
mode: 'direct' | 'batch',
|
||||
totalItems: number,
|
||||
batchesCreated?: number,
|
||||
duration: number
|
||||
}
|
||||
```
|
||||
|
||||
## Provider Migration
|
||||
|
||||
### ✅ Current Implementation
|
||||
|
||||
All providers now use the new functional approach:
|
||||
|
||||
```typescript
|
||||
'process-batch-items': async (payload: any) => {
|
||||
const { processBatchJob } = await import('../utils/batch-helpers');
|
||||
return await processBatchJob(payload, queueManager);
|
||||
}
|
||||
```
|
||||
|
||||
## Testing the New Approach
|
||||
|
||||
Use the new test endpoints:
|
||||
|
||||
```bash
|
||||
# Test symbol processing
|
||||
curl -X POST http://localhost:3002/api/test/batch-symbols \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"symbols": ["AAPL", "GOOGL"], "useBatching": false, "totalDelayMs": 10000}'
|
||||
|
||||
# Test custom processing
|
||||
curl -X POST http://localhost:3002/api/test/batch-custom \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"items": [1,2,3,4,5], "useBatching": true, "totalDelayMs": 15000}'
|
||||
```
|
||||
|
||||
## Performance Improvements
|
||||
|
||||
| Metric | Before | After | Improvement |
|
||||
|--------|--------|-------|-------------|
|
||||
| Code Lines | 545 | ~200 | 63% reduction |
|
||||
| Memory Usage | High | Low | ~40% less |
|
||||
| Initialization Time | ~2-10s | Instant | 100% faster |
|
||||
| API Complexity | High | Low | Much simpler |
|
||||
| Type Safety | Medium | High | Better types |
|
||||
|
||||
## ✅ Migration Complete
|
||||
|
||||
The old `BatchProcessor` class has been completely removed. All batch processing now uses the simplified functional approach.
|
||||
|
||||
## Common Issues & Solutions
|
||||
|
||||
### Function Serialization
|
||||
The new approach serializes processor functions for batch jobs. Avoid:
|
||||
- Closures with external variables
|
||||
- Complex function dependencies
|
||||
- Non-serializable objects
|
||||
|
||||
**Good:**
|
||||
```typescript
|
||||
(item, index) => ({ id: item.id, index })
|
||||
```
|
||||
|
||||
**Bad:**
|
||||
```typescript
|
||||
const externalVar = 'test';
|
||||
(item, index) => ({ id: item.id, external: externalVar }) // Won't work
|
||||
```
|
||||
|
||||
### Cache Dependencies
|
||||
The functional approach automatically handles cache initialization. No need to manually wait for cache readiness.
|
||||
|
||||
## Need Help?
|
||||
|
||||
Check the examples in `apps/data-ingestion/src/examples/batch-processing-examples.ts` for more detailed usage patterns.
|
||||
Loading…
Add table
Add a link
Reference in a new issue