linxus fs fixes
This commit is contained in:
parent
ac23b70146
commit
0b7846fe67
292 changed files with 41947 additions and 41947 deletions
|
|
@ -1,337 +1,337 @@
|
|||
# @stock-bot/logger
|
||||
|
||||
Enhanced logging library with Loki integration for the Stock Bot platform (June 2025).
|
||||
|
||||
## Features
|
||||
|
||||
- 🎯 **Multiple Log Levels**: debug, info, warn, error, http
|
||||
- 🌐 **Loki Integration**: Centralized logging with Grafana visualization
|
||||
- 📁 **File Logging**: Daily rotating log files with compression
|
||||
- 🎨 **Console Logging**: Colored, formatted console output
|
||||
- 📊 **Structured Logging**: JSON-formatted logs with metadata
|
||||
- ⚡ **Performance Optimized**: Batching and async logging
|
||||
- 🔐 **Security**: Automatic sensitive data masking
|
||||
- 🎭 **Express Middleware**: Request/response logging
|
||||
- 📈 **Business Events**: Specialized logging for trading operations
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Using Bun (current runtime)
|
||||
bun install
|
||||
```
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### Simple Logging
|
||||
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('my-service');
|
||||
|
||||
logger.info('Service started');
|
||||
logger.warn('This is a warning');
|
||||
logger.error('An error occurred', new Error('Something went wrong'));
|
||||
```
|
||||
|
||||
### With Context
|
||||
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('trading-service');
|
||||
|
||||
logger.info('Trade executed', {
|
||||
symbol: 'AAPL',
|
||||
quantity: 100,
|
||||
price: 150.25,
|
||||
userId: '12345',
|
||||
sessionId: 'abc-def-ghi'
|
||||
});
|
||||
```
|
||||
|
||||
### Performance Logging
|
||||
|
||||
```typescript
|
||||
import { getLogger, createTimer } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('data-processor');
|
||||
const timer = createTimer('data-processing');
|
||||
|
||||
// ... do some work ...
|
||||
|
||||
const timing = timer.end();
|
||||
logger.performance('Data processing completed', timing);
|
||||
```
|
||||
|
||||
### Business Events
|
||||
|
||||
```typescript
|
||||
import { getLogger, createBusinessEvent } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('order-service');
|
||||
|
||||
logger.business('Order placed', createBusinessEvent(
|
||||
'order',
|
||||
'place',
|
||||
{
|
||||
entity: 'order-123',
|
||||
result: 'success',
|
||||
symbol: 'TSLA',
|
||||
amount: 50000
|
||||
}
|
||||
));
|
||||
```
|
||||
|
||||
### Security Events
|
||||
|
||||
```typescript
|
||||
import { getLogger, createSecurityEvent } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('auth-service');
|
||||
|
||||
logger.security('Failed login attempt', createSecurityEvent(
|
||||
'authentication',
|
||||
{
|
||||
user: 'john@example.com',
|
||||
result: 'failure',
|
||||
ip: '192.168.1.100',
|
||||
severity: 'medium'
|
||||
}
|
||||
));
|
||||
```
|
||||
|
||||
## Express Middleware
|
||||
|
||||
### Basic Request Logging
|
||||
|
||||
```typescript
|
||||
import express from 'express';
|
||||
import { loggingMiddleware } from '@stock-bot/logger';
|
||||
|
||||
const app = express();
|
||||
|
||||
app.use(loggingMiddleware({
|
||||
serviceName: 'api-gateway',
|
||||
skipPaths: ['/health', '/metrics']
|
||||
}));
|
||||
```
|
||||
|
||||
### Error Logging
|
||||
|
||||
```typescript
|
||||
import { errorLoggingMiddleware, getLogger } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('api-gateway');
|
||||
|
||||
// Add after your routes but before error handlers
|
||||
app.use(errorLoggingMiddleware(logger));
|
||||
```
|
||||
|
||||
### Request-scoped Logger
|
||||
|
||||
```typescript
|
||||
import { createRequestLogger, getLogger } from '@stock-bot/logger';
|
||||
|
||||
const baseLogger = getLogger('api-gateway');
|
||||
|
||||
app.use((req, res, next) => {
|
||||
req.logger = createRequestLogger(req, baseLogger);
|
||||
next();
|
||||
});
|
||||
|
||||
app.get('/api/data', (req, res) => {
|
||||
req.logger.info('Processing data request');
|
||||
// ... handle request ...
|
||||
});
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
The logger uses configuration from `@stock-bot/config`. Key environment variables:
|
||||
|
||||
```bash
|
||||
# Logging
|
||||
LOG_LEVEL=info
|
||||
LOG_CONSOLE=true
|
||||
LOG_FILE=true
|
||||
LOG_FILE_PATH=./logs
|
||||
|
||||
# Loki
|
||||
LOKI_HOST=localhost
|
||||
LOKI_PORT=3100
|
||||
LOKI_BATCH_SIZE=1024
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Child Loggers
|
||||
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
const parentLogger = getLogger('trading-service');
|
||||
const orderLogger = parentLogger.child({
|
||||
module: 'order-processing',
|
||||
orderId: '12345'
|
||||
});
|
||||
|
||||
orderLogger.info('Order validated'); // Will include parent context
|
||||
```
|
||||
|
||||
### Custom Configuration
|
||||
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
// Uses standard getLogger with service-specific configuration
|
||||
const logger = getLogger('custom-service');
|
||||
```
|
||||
|
||||
### Sensitive Data Masking
|
||||
|
||||
```typescript
|
||||
import { sanitizeMetadata, maskSensitiveData } from '@stock-bot/logger';
|
||||
|
||||
const unsafeData = {
|
||||
username: 'john',
|
||||
password: 'secret123',
|
||||
apiKey: 'abc123def456'
|
||||
};
|
||||
|
||||
const safeData = sanitizeMetadata(unsafeData);
|
||||
// { username: 'john', password: '[REDACTED]', apiKey: '[REDACTED]' }
|
||||
|
||||
const message = maskSensitiveData('User API key: abc123def456');
|
||||
// 'User API key: [API_KEY]'
|
||||
```
|
||||
|
||||
### Log Throttling
|
||||
|
||||
```typescript
|
||||
import { LogThrottle } from '@stock-bot/logger';
|
||||
|
||||
const throttle = new LogThrottle(10, 60000); // 10 logs per minute
|
||||
|
||||
if (throttle.shouldLog('error-key')) {
|
||||
logger.error('This error will be throttled');
|
||||
}
|
||||
```
|
||||
|
||||
## Viewing Logs
|
||||
|
||||
### Grafana Dashboard
|
||||
|
||||
1. Start the monitoring stack: `docker-compose up grafana loki`
|
||||
2. Open Grafana at http://localhost:3000
|
||||
3. Use the "Stock Bot Logs" dashboard
|
||||
4. Query logs with LogQL: `{service="your-service"}`
|
||||
|
||||
### Log Files
|
||||
|
||||
When file logging is enabled, logs are written to:
|
||||
- `./logs/{service-name}-YYYY-MM-DD.log` - All logs
|
||||
- `./logs/{service-name}-error-YYYY-MM-DD.log` - Error logs only
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use appropriate log levels**:
|
||||
- `debug`: Detailed development information
|
||||
- `info`: General operational messages
|
||||
- `warn`: Potential issues
|
||||
- `error`: Actual errors requiring attention
|
||||
|
||||
2. **Include context**: Always provide relevant metadata
|
||||
```typescript
|
||||
logger.info('Trade executed', { symbol, quantity, price, orderId });
|
||||
```
|
||||
|
||||
3. **Use structured logging**: Avoid string concatenation
|
||||
```typescript
|
||||
// Good
|
||||
logger.info('User logged in', { userId, ip, userAgent });
|
||||
|
||||
// Avoid
|
||||
logger.info(`User ${userId} logged in from ${ip}`);
|
||||
```
|
||||
|
||||
4. **Handle sensitive data**: Use sanitization utilities
|
||||
```typescript
|
||||
const safeMetadata = sanitizeMetadata(requestData);
|
||||
logger.info('API request', safeMetadata);
|
||||
```
|
||||
|
||||
5. **Use correlation IDs**: Track requests across services
|
||||
```typescript
|
||||
const logger = getLogger('service').child({
|
||||
correlationId: req.headers['x-correlation-id']
|
||||
});
|
||||
```
|
||||
|
||||
## Integration with Services
|
||||
|
||||
To use in your service:
|
||||
|
||||
1. Add dependency to your service's `package.json`:
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"@stock-bot/logger": "*"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Update your service's `tsconfig.json` references:
|
||||
```json
|
||||
{
|
||||
"references": [
|
||||
{ "path": "../../../libs/logger" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
3. Import and use:
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('my-service');
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- Logs are batched and sent asynchronously to Loki
|
||||
- File logging uses daily rotation to prevent large files
|
||||
- Console logging can be disabled in production
|
||||
- Use log throttling for high-frequency events
|
||||
- Sensitive data is automatically masked
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Logs not appearing in Loki
|
||||
|
||||
1. Check Loki connection:
|
||||
```bash
|
||||
curl http://localhost:3100/ready
|
||||
```
|
||||
|
||||
2. Verify environment variables:
|
||||
```bash
|
||||
echo $LOKI_HOST $LOKI_PORT
|
||||
```
|
||||
|
||||
3. Check container logs:
|
||||
```bash
|
||||
docker logs stock-bot-loki
|
||||
```
|
||||
|
||||
### High memory usage
|
||||
|
||||
- Reduce `LOKI_BATCH_SIZE` if batching too many logs
|
||||
- Disable file logging if not needed
|
||||
|
||||
### Missing logs
|
||||
|
||||
- Check log level configuration
|
||||
- Verify service name matches expectations
|
||||
- Ensure proper error handling around logger calls
|
||||
# @stock-bot/logger
|
||||
|
||||
Enhanced logging library with Loki integration for the Stock Bot platform (June 2025).
|
||||
|
||||
## Features
|
||||
|
||||
- 🎯 **Multiple Log Levels**: debug, info, warn, error, http
|
||||
- 🌐 **Loki Integration**: Centralized logging with Grafana visualization
|
||||
- 📁 **File Logging**: Daily rotating log files with compression
|
||||
- 🎨 **Console Logging**: Colored, formatted console output
|
||||
- 📊 **Structured Logging**: JSON-formatted logs with metadata
|
||||
- ⚡ **Performance Optimized**: Batching and async logging
|
||||
- 🔐 **Security**: Automatic sensitive data masking
|
||||
- 🎭 **Express Middleware**: Request/response logging
|
||||
- 📈 **Business Events**: Specialized logging for trading operations
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Using Bun (current runtime)
|
||||
bun install
|
||||
```
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### Simple Logging
|
||||
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('my-service');
|
||||
|
||||
logger.info('Service started');
|
||||
logger.warn('This is a warning');
|
||||
logger.error('An error occurred', new Error('Something went wrong'));
|
||||
```
|
||||
|
||||
### With Context
|
||||
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('trading-service');
|
||||
|
||||
logger.info('Trade executed', {
|
||||
symbol: 'AAPL',
|
||||
quantity: 100,
|
||||
price: 150.25,
|
||||
userId: '12345',
|
||||
sessionId: 'abc-def-ghi'
|
||||
});
|
||||
```
|
||||
|
||||
### Performance Logging
|
||||
|
||||
```typescript
|
||||
import { getLogger, createTimer } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('data-processor');
|
||||
const timer = createTimer('data-processing');
|
||||
|
||||
// ... do some work ...
|
||||
|
||||
const timing = timer.end();
|
||||
logger.performance('Data processing completed', timing);
|
||||
```
|
||||
|
||||
### Business Events
|
||||
|
||||
```typescript
|
||||
import { getLogger, createBusinessEvent } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('order-service');
|
||||
|
||||
logger.business('Order placed', createBusinessEvent(
|
||||
'order',
|
||||
'place',
|
||||
{
|
||||
entity: 'order-123',
|
||||
result: 'success',
|
||||
symbol: 'TSLA',
|
||||
amount: 50000
|
||||
}
|
||||
));
|
||||
```
|
||||
|
||||
### Security Events
|
||||
|
||||
```typescript
|
||||
import { getLogger, createSecurityEvent } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('auth-service');
|
||||
|
||||
logger.security('Failed login attempt', createSecurityEvent(
|
||||
'authentication',
|
||||
{
|
||||
user: 'john@example.com',
|
||||
result: 'failure',
|
||||
ip: '192.168.1.100',
|
||||
severity: 'medium'
|
||||
}
|
||||
));
|
||||
```
|
||||
|
||||
## Express Middleware
|
||||
|
||||
### Basic Request Logging
|
||||
|
||||
```typescript
|
||||
import express from 'express';
|
||||
import { loggingMiddleware } from '@stock-bot/logger';
|
||||
|
||||
const app = express();
|
||||
|
||||
app.use(loggingMiddleware({
|
||||
serviceName: 'api-gateway',
|
||||
skipPaths: ['/health', '/metrics']
|
||||
}));
|
||||
```
|
||||
|
||||
### Error Logging
|
||||
|
||||
```typescript
|
||||
import { errorLoggingMiddleware, getLogger } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('api-gateway');
|
||||
|
||||
// Add after your routes but before error handlers
|
||||
app.use(errorLoggingMiddleware(logger));
|
||||
```
|
||||
|
||||
### Request-scoped Logger
|
||||
|
||||
```typescript
|
||||
import { createRequestLogger, getLogger } from '@stock-bot/logger';
|
||||
|
||||
const baseLogger = getLogger('api-gateway');
|
||||
|
||||
app.use((req, res, next) => {
|
||||
req.logger = createRequestLogger(req, baseLogger);
|
||||
next();
|
||||
});
|
||||
|
||||
app.get('/api/data', (req, res) => {
|
||||
req.logger.info('Processing data request');
|
||||
// ... handle request ...
|
||||
});
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
The logger uses configuration from `@stock-bot/config`. Key environment variables:
|
||||
|
||||
```bash
|
||||
# Logging
|
||||
LOG_LEVEL=info
|
||||
LOG_CONSOLE=true
|
||||
LOG_FILE=true
|
||||
LOG_FILE_PATH=./logs
|
||||
|
||||
# Loki
|
||||
LOKI_HOST=localhost
|
||||
LOKI_PORT=3100
|
||||
LOKI_BATCH_SIZE=1024
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Child Loggers
|
||||
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
const parentLogger = getLogger('trading-service');
|
||||
const orderLogger = parentLogger.child({
|
||||
module: 'order-processing',
|
||||
orderId: '12345'
|
||||
});
|
||||
|
||||
orderLogger.info('Order validated'); // Will include parent context
|
||||
```
|
||||
|
||||
### Custom Configuration
|
||||
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
// Uses standard getLogger with service-specific configuration
|
||||
const logger = getLogger('custom-service');
|
||||
```
|
||||
|
||||
### Sensitive Data Masking
|
||||
|
||||
```typescript
|
||||
import { sanitizeMetadata, maskSensitiveData } from '@stock-bot/logger';
|
||||
|
||||
const unsafeData = {
|
||||
username: 'john',
|
||||
password: 'secret123',
|
||||
apiKey: 'abc123def456'
|
||||
};
|
||||
|
||||
const safeData = sanitizeMetadata(unsafeData);
|
||||
// { username: 'john', password: '[REDACTED]', apiKey: '[REDACTED]' }
|
||||
|
||||
const message = maskSensitiveData('User API key: abc123def456');
|
||||
// 'User API key: [API_KEY]'
|
||||
```
|
||||
|
||||
### Log Throttling
|
||||
|
||||
```typescript
|
||||
import { LogThrottle } from '@stock-bot/logger';
|
||||
|
||||
const throttle = new LogThrottle(10, 60000); // 10 logs per minute
|
||||
|
||||
if (throttle.shouldLog('error-key')) {
|
||||
logger.error('This error will be throttled');
|
||||
}
|
||||
```
|
||||
|
||||
## Viewing Logs
|
||||
|
||||
### Grafana Dashboard
|
||||
|
||||
1. Start the monitoring stack: `docker-compose up grafana loki`
|
||||
2. Open Grafana at http://localhost:3000
|
||||
3. Use the "Stock Bot Logs" dashboard
|
||||
4. Query logs with LogQL: `{service="your-service"}`
|
||||
|
||||
### Log Files
|
||||
|
||||
When file logging is enabled, logs are written to:
|
||||
- `./logs/{service-name}-YYYY-MM-DD.log` - All logs
|
||||
- `./logs/{service-name}-error-YYYY-MM-DD.log` - Error logs only
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use appropriate log levels**:
|
||||
- `debug`: Detailed development information
|
||||
- `info`: General operational messages
|
||||
- `warn`: Potential issues
|
||||
- `error`: Actual errors requiring attention
|
||||
|
||||
2. **Include context**: Always provide relevant metadata
|
||||
```typescript
|
||||
logger.info('Trade executed', { symbol, quantity, price, orderId });
|
||||
```
|
||||
|
||||
3. **Use structured logging**: Avoid string concatenation
|
||||
```typescript
|
||||
// Good
|
||||
logger.info('User logged in', { userId, ip, userAgent });
|
||||
|
||||
// Avoid
|
||||
logger.info(`User ${userId} logged in from ${ip}`);
|
||||
```
|
||||
|
||||
4. **Handle sensitive data**: Use sanitization utilities
|
||||
```typescript
|
||||
const safeMetadata = sanitizeMetadata(requestData);
|
||||
logger.info('API request', safeMetadata);
|
||||
```
|
||||
|
||||
5. **Use correlation IDs**: Track requests across services
|
||||
```typescript
|
||||
const logger = getLogger('service').child({
|
||||
correlationId: req.headers['x-correlation-id']
|
||||
});
|
||||
```
|
||||
|
||||
## Integration with Services
|
||||
|
||||
To use in your service:
|
||||
|
||||
1. Add dependency to your service's `package.json`:
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"@stock-bot/logger": "*"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Update your service's `tsconfig.json` references:
|
||||
```json
|
||||
{
|
||||
"references": [
|
||||
{ "path": "../../../libs/logger" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
3. Import and use:
|
||||
```typescript
|
||||
import { getLogger } from '@stock-bot/logger';
|
||||
|
||||
const logger = getLogger('my-service');
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- Logs are batched and sent asynchronously to Loki
|
||||
- File logging uses daily rotation to prevent large files
|
||||
- Console logging can be disabled in production
|
||||
- Use log throttling for high-frequency events
|
||||
- Sensitive data is automatically masked
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Logs not appearing in Loki
|
||||
|
||||
1. Check Loki connection:
|
||||
```bash
|
||||
curl http://localhost:3100/ready
|
||||
```
|
||||
|
||||
2. Verify environment variables:
|
||||
```bash
|
||||
echo $LOKI_HOST $LOKI_PORT
|
||||
```
|
||||
|
||||
3. Check container logs:
|
||||
```bash
|
||||
docker logs stock-bot-loki
|
||||
```
|
||||
|
||||
### High memory usage
|
||||
|
||||
- Reduce `LOKI_BATCH_SIZE` if batching too many logs
|
||||
- Disable file logging if not needed
|
||||
|
||||
### Missing logs
|
||||
|
||||
- Check log level configuration
|
||||
- Verify service name matches expectations
|
||||
- Ensure proper error handling around logger calls
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue