Loki Logging for Stock Bot

This document outlines how to use the Loki logging system integrated with the Stock Bot platform.

Overview

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs (labels), not the full text. This makes Loki more resource-efficient than traditional log storage systems.

For Stock Bot, Loki provides:

Centralized logging for all services
Log aggregation and filtering by service, level, and custom labels
Integration with Grafana for visualization
Query capabilities for log analysis
Alert capabilities for critical issues

Getting Started

Starting the Logging Stack

# Start the monitoring stack (includes Loki and Grafana)
scripts\docker.ps1 monitoring

You can also start Loki directly using Docker Compose:

# Start Loki service only
docker-compose up -d loki

# Start Loki and Grafana
docker-compose up -d loki grafana

Viewing Logs

Once started:

Access Grafana at http://localhost:3000 (login with admin/admin)
Navigate to the "Stock Bot Logs" dashboard
View and query your logs

Using the Logger in Your Services

The Stock Bot logger has been enhanced to automatically send logs to Loki. Here's how to use it:

import { Logger, LogLevel } from '@stock-bot/utils';

// Create a logger for your service
const logger = new Logger('your-service-name', LogLevel.INFO);

// Log at different levels
logger.debug('Detailed information for debugging');
logger.info('General information about operations');
logger.warn('Potential issues that don't affect operation');
logger.error('Critical errors that require attention');

// Log with structured data (will be searchable in Loki)
logger.info('Processing trade', { 
  symbol: 'MSFT', 
  price: 410.75, 
  quantity: 50 
});

Configuration Options

Logger configuration is managed through the @stock-bot/config package and can be set in your .env file:

# Logging configuration
LOG_LEVEL=debug           # debug, info, warn, error
LOG_CONSOLE=true          # Log to console in addition to Loki
LOKI_HOST=localhost       # Loki server hostname
LOKI_PORT=3100            # Loki server port
LOKI_RETENTION_DAYS=30    # Days to retain logs
LOKI_LABELS=environment=development,service=stock-bot  # Default labels
LOKI_BATCH_SIZE=100       # Number of logs to batch before sending
LOKI_FLUSH_INTERVAL_MS=5000  # Max time to wait before sending logs

Useful Loki Queries

Inside Grafana, you can use these LogQL queries to analyze your logs:

All logs from a specific service:
```
{service="market-data-gateway"}
```
All error logs across all services:
```
{level="error"}
```

Logs containing specific text:

{service="market-data-gateway"} |= "trade"

Count of error logs by service over time:

sum by(service) (count_over_time({level="error"}[5m]))

Testing the Logging Integration

A test script is provided to verify the logging integration:

# Run from project root
bun run tools/test-loki-logging.ts

Architecture

Our logging implementation follows this architecture:

┌─────────────────┐     ┌─────────────────┐
│ Trading Services│────►│ @stock-bot/utils│
└─────────────────┘     │     Logger      │
                        └────────┬────────┘
                                 │
                                 ▼
┌────────────────────────────────────────┐
│               Loki                     │
└────────────────┬───────────────────────┘
                 │
                 ▼
┌────────────────────────────────────────┐
│              Grafana                   │
└────────────────────────────────────────┘

Adding New Dashboards

To create new Grafana dashboards for log visualization:

Build your dashboard in the Grafana UI
Export it to JSON
Add it to monitoring/grafana/provisioning/dashboards/json/
Restart the monitoring stack

Troubleshooting

If logs aren't appearing in Grafana:

Run the status check script to verify Loki and Grafana are working:
```
tools\check-loki-status.bat
```
Check that Loki and Grafana containers are running:
```
docker ps | findstr "loki grafana"
```
Verify .env configuration for Loki host and port:
```
type .env | findstr "LOKI_"
```
Ensure your service has the latest @stock-bot/utils package
Check for errors in the Loki container logs:
```
docker logs trading-bot-loki
```

5.1 KiB Raw Blame History