# Operation Management System

A unified system for managing data operations across multiple providers in the stock data ingestion pipeline.

## Overview

The Operation Management System provides a consistent way to track, schedule, and monitor data operations across different data sources (QM, CEO, IB, EOD, etc.) while maintaining provider-specific flexibility.

## Architecture

```
operation-manager/
├── BaseOperationProvider.ts    # Abstract base class for providers
├── OperationTracker.ts        # Tracks operation status per symbol
├── OperationRegistry.ts       # Central registry for all providers
└── types.ts                   # Shared type definitions

handlers/
├── qm/shared/operation-provider.ts    # QM-specific implementation
├── ceo/shared/operation-provider.ts   # CEO-specific implementation
└── ib/shared/operation-provider.ts    # IB-specific implementation
```

## Core Concepts

### Operations

Operations represent data fetching or processing tasks that need to be performed for each symbol. Each operation has:

- **Name**: Unique identifier within a provider
- **Type**: Determines tracking behavior
  - `standard`: Regular operations (prices, fundamentals)
  - `incremental`: Operations that build on previous data
  - `crawl`: Operations that work backwards through time
  - `snapshot`: Point-in-time data captures
  - `intraday_crawl`: Special crawl for intraday data (QM-specific)
- **Default Stale Hours**: When data is considered outdated
- **Priority**: For scheduling optimization

### Providers

Each data source implements a provider that:
- Defines its collection name and symbol field
- Lists all supported operations
- Can implement hooks for custom logic

### Operation Tracking

Each symbol tracks operation status including:
- Last run time
- Last successful run time
- Success/failure status
- Record count
- Last record date
- Custom metadata

## Usage

### 1. Create a Provider

```typescript
export class MyDataProvider extends BaseOperationProvider {
  getProviderConfig(): ProviderConfig {
    return {
      name: 'mydata',
      collectionName: 'myDataSymbols',
      symbolField: 'ticker'
    };
  }

  getOperations(): OperationConfig[] {
    return [
      {
        name: 'daily_prices',
        type: 'standard',
        defaultStaleHours: 24
      }
    ];
  }
}
```

### 2. Register Provider in Handler

```typescript
export class MyDataHandler extends BaseHandler {
  public operationRegistry: OperationRegistry;

  constructor(services: any) {
    super(services);
    
    this.operationRegistry = new OperationRegistry({
      mongodb: this.mongodb,
      logger: this.logger
    });
    
    const provider = new MyDataProvider({
      mongodb: this.mongodb,
      logger: this.logger
    });
    
    this.operationRegistry.registerProvider(provider);
  }
}
```

### 3. Track Operations

```typescript
// Update operation status
await this.operationRegistry.updateOperation('mydata', 'AAPL', 'daily_prices', {
  status: 'success',
  recordCount: 100,
  lastRecordDate: new Date()
});

// Get stale symbols
const staleSymbols = await this.operationRegistry.getStaleSymbols(
  'mydata', 
  'daily_prices',
  { limit: 1000 }
);

// Get operation stats
const stats = await this.operationRegistry.getOperationStats(
  'mydata',
  'daily_prices'
);
```

## Provider Examples

### QM (QuoteMedia)
- Operations: prices, financials, events, filings, intraday bars
- Symbol field: `qmSearchCode`
- Special handling for session management

### CEO (CEO.CA)
- Operations: posts, shorts, channels, sentiment
- Symbol field: `symbol`
- Tracks community data and discussions

### IB (Interactive Brokers)
- Operations: contracts, real-time quotes, historical data, options
- Symbol field: `symbol`
- Market hours awareness

## Database Schema

Operations are tracked in the provider's symbol collection:

```javascript
{
  symbol: "AAPL",
  operations: {
    price_update: {
      lastRunAt: ISODate("2024-01-15T10:00:00Z"),
      lastSuccessAt: ISODate("2024-01-15T10:00:00Z"),
      status: "success",
      recordCount: 252,
      lastRecordDate: ISODate("2024-01-14T00:00:00Z")
    },
    financials_update: {
      lastRunAt: ISODate("2024-01-10T08:00:00Z"),
      lastSuccessAt: ISODate("2024-01-10T08:00:00Z"),
      status: "success",
      recordCount: 4,
      lastRecordDate: ISODate("2023-12-31T00:00:00Z")
    }
  }
}
```

## Indexes

The system automatically creates indexes for efficient querying:
- `operations.{name}.lastSuccessAt` + symbol field
- `operations.{name}.lastRecordDate` + symbol field
- `operations.{name}.status` + symbol field
- Additional indexes for crawl operations

## Migration Guide

See [MIGRATION_GUIDE.md](./MIGRATION_GUIDE.md) for detailed steps on migrating from provider-specific tracking.

## Best Practices

1. **Operation Naming**: Use descriptive names with underscores (e.g., `price_update`, `posts_crawl`)
2. **Stale Hours**: Set appropriate defaults based on data freshness requirements
3. **Error Handling**: Always update operation status, even on failure
4. **Bulk Operations**: Use bulk updates when processing multiple symbols
5. **Metadata**: Store provider-specific data in metadata fields

## Performance Considerations

- Indexes are created automatically for common query patterns
- Use bulk operations for updating multiple symbols
- Query with appropriate limits to avoid memory issues
- Consider operation priorities for scheduling

## Future Enhancements

- Cross-provider operation dependencies
- Operation scheduling optimization
- Real-time monitoring dashboard
- Automatic retry mechanisms
- Operation result caching