SigNoz Integration

Send OpenTelemetry traces, metrics, and logs from Databrain to SigNoz (open-source observability)

Integrating Databrain with SigNoz

This guide explains how to send OpenTelemetry traces, metrics, and logs from your self-hosted Databrain instance to SigNoz, an open-source observability platform.

Why SigNoz?

SigNoz is a great choice for self-hosted observability:

• 100% Open Source: No vendor lock-in
• OpenTelemetry Native: Built specifically for OTel
• All-in-One: Traces, metrics, and logs in a single platform
• Cost-Effective: Self-hosted means no per-event pricing
• Easy to Deploy: Docker Compose or Kubernetes deployment

Prerequisites

• Databrain self-hosted version with OpenTelemetry support
• Docker and Docker Compose (for SigNoz installation)
• 4GB+ RAM for SigNoz (8GB recommended for production)

Option 1: SigNoz Cloud (Easiest)

If you prefer a managed solution, use SigNoz Cloud:

1. Sign Up for SigNoz Cloud

1. Go to https://signoz.io/teams/
2. Create a free account
3. Note your ingestion endpoint and ingestion key

2. Configure Databrain

# Enable OpenTelemetry
OTEL_ENABLED=true

# SigNoz Cloud endpoint (adjust for your region)
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.{region}.signoz.cloud:443

# SigNoz ingestion key
OTEL_EXPORTER_OTLP_HEADERS=signoz-access-token=YOUR_INGESTION_KEY

# Service name
OTEL_SERVICE_NAME=databrain-api

# Optional: Enable debug logging
LOG_LEVEL=info

3. Docker Compose Configuration

services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://ingest.{region}.signoz.cloud:443"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "signoz-access-token=${SIGNOZ_INGESTION_KEY}"
      LOG_LEVEL: "info"

Option 2: Self-Hosted SigNoz (Recommended for Production)

1. Install SigNoz

Using Docker Compose (Quick Start)

# Clone SigNoz repository
git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy/

# Start SigNoz
docker compose -f docker/clickhouse-setup/docker-compose.yaml up -d

This starts:

• SigNoz UI: http://localhost:3301
• OTLP Receiver (gRPC): localhost:4317
• OTLP Receiver (HTTP): localhost:4318
• ClickHouse: For data storage
• Query Service: For querying data

Using Docker Compose (Custom Network)

If Databrain is on the same Docker network:

# signoz-docker-compose.yaml
version: "3.8"

networks:
  databrain:
    external: true
  signoz:
    name: signoz

services:
  clickhouse:
    image: clickhouse/clickhouse-server:23.7.3-alpine
    container_name: signoz-clickhouse
    volumes:
      - ./clickhouse-config.xml:/etc/clickhouse-server/config.d/config.xml
      - ./clickhouse-users.xml:/etc/clickhouse-server/users.d/users.xml
      - signoz-data:/var/lib/clickhouse/
    networks:
      - signoz
    restart: on-failure

  otel-collector:
    image: signoz/signoz-otel-collector:0.88.11
    container_name: signoz-otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"  # OTLP gRPC receiver
      - "4318:4318"  # OTLP HTTP receiver
    networks:
      - signoz
      - databrain
    depends_on:
      - clickhouse
    restart: on-failure

  query-service:
    image: signoz/query-service:0.38.0
    container_name: signoz-query-service
    command: ["-config=/root/config/prometheus.yml"]
    volumes:
      - ./prometheus.yml:/root/config/prometheus.yml
    environment:
      - ClickHouseUrl=tcp://clickhouse:9000
    networks:
      - signoz
    depends_on:
      - clickhouse
    restart: on-failure

  frontend:
    image: signoz/frontend:0.38.0
    container_name: signoz-frontend
    ports:
      - "3301:3301"
    environment:
      - FRONTEND_API_ENDPOINT=http://query-service:8080
    networks:
      - signoz
    depends_on:
      - query-service
    restart: on-failure

volumes:
  signoz-data:

2. Configure Databrain for Self-Hosted SigNoz

Same Docker Network

# docker-compose.yml
services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://signoz-otel-collector:4318"
      OTEL_SERVICE_NAME: "databrain-api"
      LOG_LEVEL: "info"
    networks:
      - databrain
      - signoz

networks:
  databrain:
  signoz:
    external: true

Separate Hosts

If SigNoz is on a different host:

# Enable OpenTelemetry
OTEL_ENABLED=true

# SigNoz OTLP endpoint (adjust host)
OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-host:4318

# Service name
OTEL_SERVICE_NAME=databrain-api

3. Kubernetes Deployment

Deploy SigNoz on Kubernetes

# Add SigNoz Helm repo
helm repo add signoz https://charts.signoz.io
helm repo update

# Create namespace
kubectl create namespace signoz

# Install SigNoz
helm install signoz signoz/signoz -n signoz \
  --set clickhouse.persistence.size=50Gi \
  --set frontend.ingress.enabled=true \
  --set frontend.ingress.hosts[0].host=signoz.your-domain.com

Configure Databrain

apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://signoz-otel-collector.signoz.svc.cluster.local:4318"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: LOG_LEVEL
            value: "info"

What Gets Sent to SigNoz

Telemetry Type	Description
Traces	Full distributed traces with all spans
Metrics	Request rates, latency histograms, error rates
Logs	Structured logs with trace correlation

All three are integrated in SigNoz’s unified interface.

Verification

1. Check SigNoz is Running

# Check containers
docker ps | grep signoz

# Check OTLP receiver is listening
curl http://localhost:4318/v1/traces
# Should return 405 Method Not Allowed (endpoint exists but expects POST)

2. Restart Databrain

docker compose restart databrainbackend

3. Generate Test Traffic

# Health check
curl https://your-databrain-instance.com/api/health

# Sample request
curl -X POST "https://your-databrain-instance.com/api/v2/metric/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"metricId": "test-123"}'

4. View in SigNoz UI

1. Open SigNoz: http://localhost:3301 (or your configured URL)
2. Go to Services → You should see databrain-api
3. Click on service to see metrics:
   • Request rate (requests per second)
   • Error rate (%)
   • Latency (p50, p90, p95, p99)
   • Apdex score
4. Go to Traces → Filter by serviceName=databrain-api
5. Click on any trace to see the full waterfall view

5. Check Logs

1. Go to Logs tab
2. Filter: service_name = databrain-api
3. Click on any log entry
4. See Related Traces section for correlated traces

SigNoz Features

1. Service Dashboard

The service dashboard shows:

• RED metrics: Rate, Errors, Duration
• Apdex score: User satisfaction metric
• Operations: Individual endpoint performance
• Database calls: Query performance breakdown
• External calls: Downstream service latency

2. Traces Explorer

Advanced trace filtering:

serviceName = databrain-api
AND duration > 1000ms
AND statusCode >= 500

Available filters:

• Service name
• Operation name
• HTTP status code
• Duration
• Error status
• Custom tags (userId, metricId, etc.)

3. Service Map

Visualize service dependencies:

1. Go to Service Map
2. See databrain-api and its dependencies:
   • PostgreSQL
   • Redis
   • Hasura
   • Keycloak
   • S3
3. Click on connections to see request rates and error rates

4. Logs Management

SigNoz provides powerful log querying:

Query examples:

service_name = databrain-api AND level = error
service_name = databrain-api AND userId = 123
service_name = databrain-api AND duration > 1000

Log to trace correlation:

• Click on any log → see Trace button
• Jump directly to the trace that generated this log

5. Alerts

Create alerts in SigNoz:

High Error Rate Alert

1. Go to Alerts → New Alert
2. Alert Type: Metrics Alert
3. Metric Query:

A: error_count = count(signoz_latency_bucket{service_name="databrain-api", status_code=~"5.."})
B: total_count = count(signoz_latency_bucket{service_name="databrain-api"})
Formula: (A / B) * 100

4. Condition: > 5 (5% error rate)
5. For: 5 minutes
6. Add notification channel (Slack, email, webhook)

High Latency Alert

1. Metric Query:

histogram_quantile(0.95, 
  sum by (le)(rate(signoz_latency_bucket{service_name="databrain-api"}[5m]))
)

2. Condition: > 2000 (2 seconds)
3. For: 5 minutes

6. Dashboards

Create custom dashboards:

1. Go to Dashboards → New Dashboard
2. Add panels:

Request Rate Panel:

rate(signoz_calls_total{service_name="databrain-api"}[5m])

Error Rate Panel:

rate(signoz_calls_total{service_name="databrain-api", status_code=~"5.."}[5m])
/
rate(signoz_calls_total{service_name="databrain-api"}[5m])

Latency Distribution Panel:

histogram_quantile(0.50, sum(rate(signoz_latency_bucket[5m])) by (le))
histogram_quantile(0.95, sum(rate(signoz_latency_bucket[5m])) by (le))
histogram_quantile(0.99, sum(rate(signoz_latency_bucket[5m])) by (le))

Database Query Latency:

histogram_quantile(0.95,
  sum(rate(signoz_latency_bucket{
    service_name="databrain-api",
    db_system="postgresql"
  }[5m])) by (le)
)

Advanced Configuration

Sampling

Add sampling to the OTel collector config:

# otel-collector-config.yaml
processors:
  # Tail-based sampling
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    policies:
      # Always keep errors
      - name: error-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      
      # Keep slow requests
      - name: latency-policy
        type: latency
        latency:
          threshold_ms: 1000
      
      # Sample 10% of normal traffic
      - name: probabilistic-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [clickhouse]

Resource Attributes

Add custom resource attributes:

processors:
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
      - key: cloud.provider
        value: aws
        action: upsert
      - key: cloud.region
        value: us-east-1
        action: upsert

Troubleshooting

Issue	Solution
No data in SigNoz	Check OTel collector logs: docker logs signoz-otel-collector
Cannot connect to SigNoz	Verify networks and firewall rules
Missing logs	Ensure Winston is configured with JSON format
High disk usage	Adjust ClickHouse retention settings
Slow queries	Increase ClickHouse resources or add indexes

Debug SigNoz Collector

# Check collector logs
docker logs signoz-otel-collector -f

# Check for OTLP receiver
docker logs signoz-otel-collector | grep "OTLP receiver"

# Should see:
# OTLP receiver started on 0.0.0.0:4317
# OTLP receiver started on 0.0.0.0:4318

Debug Databrain Connection

# Test connectivity from Databrain container
docker exec databrain-backend curl -v http://signoz-otel-collector:4318/v1/traces

Check ClickHouse

# Connect to ClickHouse
docker exec -it signoz-clickhouse clickhouse-client

# Check traces table
SELECT count() FROM signoz_traces.signoz_index_v2;

# Check for recent traces
SELECT timestamp, serviceName, name FROM signoz_traces.signoz_index_v2 
ORDER BY timestamp DESC LIMIT 10;

Data Retention

Configure retention in ClickHouse:

-- Connect to ClickHouse
docker exec -it signoz-clickhouse clickhouse-client

-- Set 30-day retention for traces
ALTER TABLE signoz_traces.signoz_index_v2 
MODIFY TTL toDateTime(timestamp) + INTERVAL 30 DAY;

-- Set 90-day retention for metrics
ALTER TABLE signoz_metrics.distributed_samples_v2 
MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;

Or configure in clickhouse-config.xml:

<clickhouse>
  <merge_tree>
    <ttl_only_drop_parts>1</ttl_only_drop_parts>
  </merge_tree>
</clickhouse>

Performance Tuning

For High Traffic (>10K requests/minute)

1. Increase ClickHouse resources:

clickhouse:
  resources:
    limits:
      memory: 8Gi
      cpu: 4

2. Batch processing:

processors:
  batch:
    timeout: 10s
    send_batch_size: 10000

3. Use buffer:

processors:
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
    check_interval: 5s

Backup and Restore

Backup ClickHouse Data

# Stop SigNoz
docker compose down

# Backup data directory
tar -czf signoz-backup-$(date +%Y%m%d).tar.gz \
  ./deploy/docker/clickhouse-setup/data/

# Restart SigNoz
docker compose up -d

Restore from Backup

# Stop SigNoz
docker compose down

# Restore data
tar -xzf signoz-backup-20260203.tar.gz -C ./deploy/docker/clickhouse-setup/

# Restart
docker compose up -d

Upgrading SigNoz

cd signoz/deploy/

# Pull latest images
docker compose -f docker/clickhouse-setup/docker-compose.yaml pull

# Restart with new images
docker compose -f docker/clickhouse-setup/docker-compose.yaml up -d

Community & Support

• Documentation: https://signoz.io/docs/
• Slack Community: https://signoz.io/slack
• GitHub: https://github.com/SigNoz/signoz
• GitHub Discussions: https://github.com/SigNoz/signoz/discussions

For Databrain configuration issues, contact your Databrain support team.