boki/stock-bot
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

switched proxy and batch to new redis

Browse source
This commit is contained in:
Boki 2025-06-10 13:52:06 -04:00
parent e98b1d8ae2
commit a86367bec5
3 changed files with 29 additions and 521 deletions
Download patch file Download diff file Expand all files Collapse all files

5
apps/data-service/src/providers/proxy.tasks.ts
View file

@ -1,5 +1,5 @@
import { getLogger } from '@stock-bot/logger'; import { getLogger } from '@stock-bot/logger';
import createCache, { type CacheProvider } from '@stock-bot/cache'; import { createCache, type CacheProvider } from '@stock-bot/cache';
import { HttpClient, ProxyInfo } from '@stock-bot/http'; import { HttpClient, ProxyInfo } from '@stock-bot/http';
import pLimit from 'p-limit'; import pLimit from 'p-limit';
@ -110,8 +110,7 @@ async function resetProxyStats(): Promise<void> {
async function initializeSharedResources() { async function initializeSharedResources() {
if (!logger) { if (!logger) {
logger = getLogger('proxy-tasks'); logger = getLogger('proxy-tasks');
cache = createCache('hybrid', { cache = createCache({
name: 'proxy-tasks',
keyPrefix: 'proxy:', keyPrefix: 'proxy:',
ttl: PROXY_CONFIG.CACHE_TTL, ttl: PROXY_CONFIG.CACHE_TTL,
enableMetrics: true enableMetrics: true

6
apps/data-service/src/utils/batch-processor.ts
View file

@ -27,10 +27,8 @@ export class BatchProcessor {
private queueManager: any, private queueManager: any,
private cacheOptions?: { keyPrefix?: string; ttl?: number } // Optional cache configuration private cacheOptions?: { keyPrefix?: string; ttl?: number } // Optional cache configuration
) { ) {
this.keyPrefix = cacheOptions?.keyPrefix || 'batch:'; this.keyPrefix = cacheOptions?.keyPrefix || 'batch:'; // Initialize cache provider with batch-specific settings
// Initialize cache provider with batch-specific settings this.cacheProvider = createCache({
this.cacheProvider = createCache('redis', {
name: 'batch-processor',
keyPrefix: this.keyPrefix, keyPrefix: this.keyPrefix,
ttl: cacheOptions?.ttl || 86400 * 2, // 48 hours default ttl: cacheOptions?.ttl || 86400 * 2, // 48 hours default
enableMetrics: true enableMetrics: true

539
docs/cache-library-usage.md
View file

@ -1,532 +1,43 @@
# Cache Library Usage Guide # Cache Library Usage Guide
The `@stock-bot/cache` library provides a powerful, flexible caching solution designed specifically for trading bot applications. It supports multiple cache providers including Redis/Dragonfly, in-memory caching, and hybrid caching strategies. > **⚠️ DEPRECATED**: This documentation is outdated. The cache library has been simplified to only use Redis/Dragonfly.
>
> **Please see [simplified-cache-usage.md](./simplified-cache-usage.md) for current usage instructions.**
## Table of Contents The `@stock-bot/cache` library now provides a simplified Redis-only caching solution designed specifically for trading bot applications.
1. [Installation](#installation) ## Migration from Old API
2. [Quick Start](#quick-start)
3. [Cache Providers](#cache-providers)
4. [Factory Functions](#factory-functions)
5. [Cache Decorators](#cache-decorators)
6. [Trading-Specific Usage](#trading-specific-usage)
7. [Configuration](#configuration)
8. [Monitoring & Metrics](#monitoring--metrics)
9. [Error Handling](#error-handling)
10. [Best Practices](#best-practices)
## Installation If you're migrating from the old cache API, here are the key changes:
The cache library is already included in the monorepo. To use it in your service:
```json
{
"dependencies": {
"@stock-bot/cache": "*"
}
}
```
## Quick Start
### Basic Usage
### Old API (DEPRECATED)
```typescript ```typescript
import { createCache } from '@stock-bot/cache'; // These are no longer supported
// Auto-detect best cache type (hybrid if Redis available, otherwise memory)
const cache = createCache('auto'); const cache = createCache('auto');
const cache = createCache('memory');
// Basic operations
await cache.set('user:123', { name: 'John', balance: 1000 }, 3600);
const user = await cache.get<{ name: string; balance: number }>('user:123');
await cache.delete('user:123');
```
### Trading-Optimized Cache
```typescript
import { createTradingCache } from '@stock-bot/cache';
const cache = createTradingCache({
keyPrefix: 'trading:',
ttl: 300, // 5 minutes default
enableMetrics: true
});
// Cache market data
await cache.set('market:AAPL:price', { price: 150.25, timestamp: Date.now() });
```
## Cache Providers
### 1. Redis Cache (Dragonfly)
Uses Redis/Dragonfly for distributed caching with persistence and high performance.
```typescript
import { RedisCache } from '@stock-bot/cache';
const redisCache = new RedisCache({
keyPrefix: 'app:',
ttl: 3600,
enableMetrics: true
});
// Automatic connection to Dragonfly using config
await redisCache.set('key', 'value');
```
### 2. Memory Cache
In-memory caching with LRU eviction and TTL support.
```typescript
import { MemoryCache } from '@stock-bot/cache';
const memoryCache = new MemoryCache({
maxSize: 1000, // Maximum 1000 entries
ttl: 300, // 5 minutes default TTL
cleanupInterval: 60 // Cleanup every minute
});
```
### 3. Hybrid Cache
Two-tier caching combining fast memory cache (L1) with persistent Redis cache (L2).
```typescript
import { HybridCache } from '@stock-bot/cache';
const hybridCache = new HybridCache({
memoryTTL: 60, // L1 cache TTL: 1 minute
redisTTL: 3600, // L2 cache TTL: 1 hour
memoryMaxSize: 500 // L1 cache max entries
});
// Data flows: Memory -> Redis -> Database
const data = await hybridCache.get('expensive:calculation');
```
## Factory Functions
### createCache()
General-purpose cache factory with auto-detection.
```typescript
import { createCache } from '@stock-bot/cache';
// Auto-detect (recommended)
const cache = createCache('auto');
// Specific provider
const redisCache = createCache('redis', { ttl: 1800 });
const memoryCache = createCache('memory', { maxSize: 2000 });
const hybridCache = createCache('hybrid');
```
### createTradingCache()
Optimized for trading operations with sensible defaults.
```typescript
import { createTradingCache } from '@stock-bot/cache';
const tradingCache = createTradingCache({
keyPrefix: 'trading:',
ttl: 300, // 5 minutes - good for price data
enableMetrics: true
});
```
### createMarketDataCache()
Specialized for market data with short TTLs.
```typescript
import { createMarketDataCache } from '@stock-bot/cache';
const marketCache = createMarketDataCache({
priceDataTTL: 30, // 30 seconds for price data
indicatorDataTTL: 300, // 5 minutes for indicators
newsDataTTL: 1800 // 30 minutes for news
});
```
### createStrategyCache()
For strategy computations and backtesting results.
```typescript
import { createStrategyCache } from '@stock-bot/cache';
const strategyCache = createStrategyCache({
backtestTTL: 86400, // 24 hours for backtest results
signalTTL: 300, // 5 minutes for signals
optimizationTTL: 3600 // 1 hour for optimization results
});
```
## Cache Decorators
### @Cacheable
Automatically cache method results.
```typescript
import { Cacheable } from '@stock-bot/cache';
class MarketDataService {
@Cacheable({
keyGenerator: (symbol: string) => `price:${symbol}`,
ttl: 60
})
async getPrice(symbol: string): Promise<number> {
// Expensive API call
return await this.fetchPriceFromAPI(symbol);
}
}
```
### @CacheEvict
Invalidate cache entries when data changes.
```typescript
import { CacheEvict } from '@stock-bot/cache';
class PortfolioService {
@CacheEvict({
keyPattern: 'portfolio:*'
})
async updatePosition(symbol: string, quantity: number): Promise<void> {
// Update database
await this.savePosition(symbol, quantity);
// Cache automatically invalidated
}
}
```
### @CachePut
Always execute method and update cache.
```typescript
import { CachePut } from '@stock-bot/cache';
class StrategyService {
@CachePut({
keyGenerator: (strategyId: string) => `strategy:${strategyId}:result`
})
async runStrategy(strategyId: string): Promise<StrategyResult> {
const result = await this.executeStrategy(strategyId);
// Result always cached after execution
return result;
}
}
```
## Trading-Specific Usage
### Market Data Caching
```typescript
import { createMarketDataCache, CacheKeyGenerator } from '@stock-bot/cache';
const marketCache = createMarketDataCache();
const keyGen = new CacheKeyGenerator();
// Cache price data
const priceKey = keyGen.priceKey('AAPL');
await marketCache.set(priceKey, { price: 150.25, volume: 1000000 }, 30);
// Cache technical indicators
const smaKey = keyGen.indicatorKey('AAPL', 'SMA', { period: 20 });
await marketCache.set(smaKey, 148.50, 300);
// Cache order book
const orderBookKey = keyGen.orderBookKey('AAPL');
await marketCache.set(orderBookKey, orderBookData, 5);
```
### Strategy Result Caching
```typescript
import { createStrategyCache, CacheKeyGenerator } from '@stock-bot/cache';
const strategyCache = createStrategyCache();
const keyGen = new CacheKeyGenerator();
// Cache backtest results
const backtestKey = keyGen.backtestKey('momentum-strategy', {
startDate: '2024-01-01',
endDate: '2024-12-31',
symbol: 'AAPL'
});
await strategyCache.set(backtestKey, backtestResults, 86400);
// Cache trading signals
const signalKey = keyGen.signalKey('AAPL', 'momentum-strategy');
await strategyCache.set(signalKey, { action: 'BUY', confidence: 0.85 }, 300);
```
### Portfolio Data Caching
```typescript
import { createTradingCache, CacheKeyGenerator } from '@stock-bot/cache';
const portfolioCache = createTradingCache();
const keyGen = new CacheKeyGenerator();
// Cache portfolio positions
const positionsKey = keyGen.portfolioKey('user123', 'positions');
await portfolioCache.set(positionsKey, positions, 300);
// Cache risk metrics
const riskKey = keyGen.riskKey('user123', 'VaR');
await portfolioCache.set(riskKey, { var95: 1250.50 }, 3600);
```
## Configuration
Cache configuration is handled through the `@stock-bot/config` package. Key settings:
```typescript
// Dragonfly/Redis configuration
DRAGONFLY_HOST=localhost
DRAGONFLY_PORT=6379
DRAGONFLY_PASSWORD=your_password
DRAGONFLY_DATABASE=0
DRAGONFLY_MAX_RETRIES=3
DRAGONFLY_RETRY_DELAY=100
DRAGONFLY_CONNECT_TIMEOUT=10000
DRAGONFLY_COMMAND_TIMEOUT=5000
// TLS settings (optional)
DRAGONFLY_TLS=true
DRAGONFLY_TLS_CERT_FILE=/path/to/cert.pem
DRAGONFLY_TLS_KEY_FILE=/path/to/key.pem
DRAGONFLY_TLS_CA_FILE=/path/to/ca.pem
```
## Monitoring & Metrics
### Cache Statistics
```typescript
const cache = createTradingCache({ enableMetrics: true });
// Get cache statistics
const stats = await cache.getStats();
console.log(`Hit rate: ${stats.hitRate}%`);
console.log(`Total operations: ${stats.total}`);
console.log(`Uptime: ${stats.uptime} seconds`);
```
### Health Checks
```typescript
const cache = createCache('hybrid'); const cache = createCache('hybrid');
const cache = createCache('redis');
// Check cache health // Direct imports no longer available
const isHealthy = await cache.isHealthy(); import { MemoryCache, HybridCache } from '@stock-bot/cache';
if (!isHealthy) {
console.error('Cache is not healthy');
}
// Monitor connection status
cache.on('connect', () => console.log('Cache connected'));
cache.on('disconnect', () => console.error('Cache disconnected'));
cache.on('error', (error) => console.error('Cache error:', error));
``` ```
### Metrics Integration ### New Simplified API
```typescript ```typescript
// Export metrics to Prometheus/Grafana // Use factory functions with options only
const metrics = await cache.getStats(); const cache = createCache({ keyPrefix: 'app:', ttl: 3600 });
const tradingCache = createTradingCache();
const marketCache = createMarketDataCache();
// Custom metrics tracking // Only Redis cache is available
await cache.set('key', 'value', 300, { import { RedisCache, RedisConnectionManager } from '@stock-bot/cache';
tags: { service: 'trading-bot', operation: 'price-update' }
});
``` ```
## Error Handling ## Quick Migration Steps
The cache library implements graceful error handling: 1. **Remove cache type parameters**: Change `createCache('hybrid')` to `createCache()`
2. **Remove name parameter**: The `name` option is no longer needed
3. **Update imports**: Use named imports instead of default import
4. **Use specialized factories**: Consider using `createTradingCache()`, `createMarketDataCache()`, etc.
### Automatic Failover For complete usage examples and best practices, see [simplified-cache-usage.md](./simplified-cache-usage.md).
```typescript
// Hybrid cache automatically falls back to memory if Redis fails
const hybridCache = createCache('hybrid');
// If Redis is down, data is served from memory cache
const data = await hybridCache.get('key'); // Never throws, returns null if not found
```
### Circuit Breaker Pattern
```typescript
const cache = createTradingCache({
maxConsecutiveFailures: 5, // Open circuit after 5 failures
circuitBreakerTimeout: 30000 // Try again after 30 seconds
});
try {
await cache.set('key', 'value');
} catch (error) {
// Handle cache unavailability
console.warn('Cache unavailable, falling back to direct data access');
}
```
### Error Events
```typescript
cache.on('error', (error) => {
if (error.code === 'CONNECTION_LOST') {
// Handle connection loss
await cache.reconnect();
}
});
```
## Best Practices
### 1. Choose the Right Cache Type
- **Memory Cache**: Fast access, limited by RAM, good for frequently accessed small data
- **Redis Cache**: Persistent, distributed, good for shared data across services
- **Hybrid Cache**: Best of both worlds, use for hot data with fallback
### 2. Set Appropriate TTLs
```typescript
// Trading data TTL guidelines
const TTL = {
PRICE_DATA: 30, // 30 seconds - very volatile
INDICATORS: 300, // 5 minutes - calculated values
NEWS: 1800, // 30 minutes - slower changing
BACKTEST_RESULTS: 86400, // 24 hours - expensive calculations
USER_PREFERENCES: 3600 // 1 hour - rarely change during session
};
```
### 3. Use Proper Key Naming
```typescript
// Good key naming convention
const keyGen = new CacheKeyGenerator();
const key = keyGen.priceKey('AAPL'); // trading:price:AAPL:2024-01-01
// Avoid generic keys
// Bad: "data", "result", "temp"
// Good: "trading:price:AAPL", "strategy:momentum:signals"
```
### 4. Implement Cache Warming
```typescript
// Pre-populate cache with frequently accessed data
async function warmupCache() {
const symbols = ['AAPL', 'GOOGL', 'MSFT'];
const cache = createMarketDataCache();
for (const symbol of symbols) {
const price = await fetchPrice(symbol);
await cache.set(keyGen.priceKey(symbol), price, 300);
}
}
```
### 5. Monitor Cache Performance
```typescript
// Regular performance monitoring
setInterval(async () => {
const stats = await cache.getStats();
if (stats.hitRate < 80) {
console.warn('Low cache hit rate:', stats.hitRate);
}
}, 60000); // Check every minute
```
### 6. Handle Cache Invalidation
```typescript
// Invalidate related cache entries when data changes
class PositionService {
async updatePosition(symbol: string, quantity: number) {
await this.saveToDatabase(symbol, quantity);
// Invalidate related cache entries
await cache.delete(`portfolio:positions`);
await cache.delete(`portfolio:risk:*`);
await cache.delete(`strategy:signals:${symbol}`);
}
}
```
## Advanced Examples
### Custom Cache Provider
```typescript
class DatabaseCache implements CacheProvider {
async get<T>(key: string): Promise<T | null> {
// Implement database-backed cache
}
async set<T>(key: string, value: T, ttl?: number): Promise<boolean> {
// Store in database with expiration
}
// ... implement other methods
}
// Use with factory
const dbCache = new DatabaseCache();
```
### Batch Operations
```typescript
// Efficient batch operations
const keys = ['price:AAPL', 'price:GOOGL', 'price:MSFT'];
const values = await cache.mget<number>(keys);
const updates = new Map([
['price:AAPL', 150.25],
['price:GOOGL', 2800.50],
['price:MSFT', 350.75]
]);
await cache.mset(updates, 300);
```
### Conditional Caching
```typescript
class SmartCache {
async getOrCompute<T>(
key: string,
computeFn: () => Promise<T>,
shouldCache: (value: T) => boolean = () => true
): Promise<T> {
let value = await this.cache.get<T>(key);
if (value === null) {
value = await computeFn();
if (shouldCache(value)) {
await this.cache.set(key, value, this.defaultTTL);
}
}
return value;
}
}
```
This cache library provides enterprise-grade caching capabilities specifically designed for trading bot applications, with built-in monitoring, error handling, and performance optimization.