Observability Setup

Configure logging and OpenTelemetry for monitoring your self-hosted Databrain instance

Observability Setup for Self-Hosted Deployments

This guide explains how to configure logging levels and enable OpenTelemetry for comprehensive observability in your self-hosted Databrain deployment.

Log Level Configuration

Databrain supports configurable log levels to help with troubleshooting. By default, the log level is set to info.

Available Log Levels

Level	Description
error	Only error messages
warn	Warnings and errors
info	Informational messages, warnings, and errors (default)
debug	All messages including detailed debug information

Configuration

Add to your .env file:

LOG_LEVEL=info

To enable debug logging for troubleshooting:

LOG_LEVEL=debug

OpenTelemetry Integration

Databrain supports OpenTelemetry (OTel) for distributed tracing and metrics. This enables you to:

• Track API latency for all endpoints
• Monitor error rates
• See processing time breakdowns (database queries, external API calls, etc.)
• Correlate logs with traces for easier debugging

Prerequisites

To use OpenTelemetry, you need an OTLP-compatible collector or backend. Popular options include:

• Self-hosted: SigNoz, Jaeger, Grafana Tempo
• Cloud: Datadog, New Relic, Honeycomb, or any OTLP-compatible backend

Configuration

Add these environment variables to your .env file:

# Enable OpenTelemetry (disabled by default)
OTEL_ENABLED=true

# Your OTLP collector endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=http://your-collector:4318

# Service name for traces (optional, defaults to databrain-api)
OTEL_SERVICE_NAME=databrain-api

Docker Compose Setup

If you want to run an OpenTelemetry Collector alongside Databrain, uncomment the collector service in your docker-compose.yml:

otel-collector:
  image: otel/opentelemetry-collector-contrib:latest
  restart: always
  ports:
    - "4317:4317"   # OTLP gRPC receiver
    - "4318:4318"   # OTLP HTTP receiver
  volumes:
    - ./otel-config.yaml:/etc/otelcol/config.yaml
  networks:
    - datab

Then update your backend environment:

OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

Example: Collector Configuration for Jaeger

Create an otel-config.yaml file:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

What Gets Instrumented

When OpenTelemetry is enabled, Databrain automatically instruments:

Component	What’s Tracked
HTTP Requests	Incoming API requests with latency, status codes
Express Routes	Route-level timing and errors
PostgreSQL	Database query duration and errors
Redis	Cache operations timing
Outgoing HTTP	External API calls (Hasura, Keycloak, etc.)

Example Trace

A typical API request trace shows:

GET /api/v2/metric/execute (1,247ms)
├── middleware.auth (12ms)
│   └── HTTP POST keycloak/auth/verify (10ms)
├── getMetricConfig (45ms)
│   └── HTTP POST hasura/v1/graphql (43ms)
├── buildQuery (8ms)
├── executeQuery (1,150ms)
│   └── PostgreSQL Query (1,148ms)
└── formatResponse (32ms)

Troubleshooting

Logs Not Appearing at Expected Level

1. Verify LOG_LEVEL is set correctly in your .env
2. Restart the backend service after changing the value
3. Log levels are case-insensitive (DEBUG, debug, Debug all work)

OpenTelemetry Not Working

1. Verify OTEL_ENABLED=true (must be exactly true)
2. Check that OTEL_EXPORTER_OTLP_ENDPOINT is reachable from the backend container
3. Check backend logs for [Telemetry] OpenTelemetry initialized message
4. If you see OTEL_ENABLED=true but OTEL_EXPORTER_OTLP_ENDPOINT not set, configure the endpoint

High Overhead Concerns

OpenTelemetry adds minimal overhead when enabled. If you experience performance issues:

1. Ensure your collector can handle the volume
2. Consider sampling in the collector configuration
3. Disable OTel temporarily by setting OTEL_ENABLED=false

Structured Logging

Databrain uses Winston for structured, JSON-formatted logging. All logs include:

• Timestamp: ISO format with milliseconds
• Level: error, warn, info, or debug
• Message: Human-readable description
• Metadata: Structured data as key-value pairs
• Trace Context: When OpenTelemetry is enabled, includes trace_id and span_id

Example Log Output

{
  "level": "info",
  "message": "API request completed",
  "timestamp": "2026-02-03 10:30:45.123",
  "method": "POST",
  "url": "/api/v2/metric/execute",
  "statusCode": 200,
  "duration": 1247,
  "userId": "123",
  "trace_id": "1a2b3c4d5e6f7g8h",
  "span_id": "9i0j1k2l3m4n"
}

Usage in Code

import logger from 'utils/logger';

// Good - structured logging
logger.info('User logged in', { userId: '123', method: 'oauth' });
logger.error('Database error', { error: err.message, query: sqlQuery });

// Avoid - unstructured logging
console.log('User 123 logged in'); // ❌ Don't use console.log

Default Behavior (No Configuration Required)

If you don’t configure any observability settings:

• Log level defaults to info
• OpenTelemetry is disabled (zero overhead)
• Logs output to stdout in JSON format
• Structured logging with Winston is always enabled

This maintains backward compatibility with existing deployments.