Honeycomb Integration

Send OpenTelemetry traces, metrics, and logs from Databrain to Honeycomb

Integrating Databrain with Honeycomb

This guide explains how to send OpenTelemetry traces and events from your self-hosted Databrain instance to Honeycomb.

Why Honeycomb?

Honeycomb is purpose-built for OpenTelemetry and excels at:

• High-cardinality data: Query on any attribute combination
• BubbleUp: Automatically surface anomalies and outliers
• Distributed tracing: Full request lifecycle visibility
• Real-time analysis: Sub-second query performance

Prerequisites

• Databrain self-hosted version with OpenTelemetry support
• Honeycomb account (free tier available)
• Honeycomb API key

Configuration

1. Get Your Honeycomb API Key

1. Log into Honeycomb
2. Go to Account → Team Settings → API Keys
3. Create a new API key or copy an existing one
4. Note your dataset name (or create a new one)

2. Configure Databrain Environment Variables

Honeycomb supports OpenTelemetry natively, making configuration simple:

# Enable OpenTelemetry
OTEL_ENABLED=true

# Honeycomb OTLP endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443

# Honeycomb API key
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=YOUR_API_KEY_HERE

# Service name (appears in Honeycomb)
OTEL_SERVICE_NAME=databrain-api

# Optional: Honeycomb dataset (defaults to service name)
HONEYCOMB_DATASET=databrain-production

# Optional: Enable debug logging
LOG_LEVEL=info

3. Docker Compose Configuration

Update your docker-compose.yml:

services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.honeycomb.io:443"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "x-honeycomb-team=${HONEYCOMB_API_KEY}"
      HONEYCOMB_DATASET: "databrain-production"
      LOG_LEVEL: "info"

Security: Store your API key in .env:

# .env
HONEYCOMB_API_KEY=your_honeycomb_api_key_here

4. Kubernetes Configuration

apiVersion: v1
kind: Secret
metadata:
  name: honeycomb-secret
type: Opaque
stringData:
  api-key: your_honeycomb_api_key_here
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "https://api.honeycomb.io:443"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: HONEYCOMB_API_KEY
            valueFrom:
              secretKeyRef:
                name: honeycomb-secret
                key: api-key
          - name: OTEL_EXPORTER_OTLP_HEADERS
            value: "x-honeycomb-team=$(HONEYCOMB_API_KEY)"
          - name: HONEYCOMB_DATASET
            value: "databrain-production"

Alternative: Using OpenTelemetry Collector

For advanced features like sampling and enrichment:

Docker Compose with Collector

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
    environment:
      HONEYCOMB_API_KEY: "${HONEYCOMB_API_KEY}"
      HONEYCOMB_DATASET: "databrain-production"
    networks:
      - databrain

  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"
      OTEL_SERVICE_NAME: "databrain-api"
    depends_on:
      - otel-collector

Collector Configuration for Honeycomb

Create otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  
  # Add resource attributes
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
      - key: service.version
        value: 1.0.0
        action: upsert
  
  # Tail-based sampling (only send interesting traces)
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    policies:
      # Always sample errors
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      
      # Sample slow requests
      - name: slow-requests
        type: latency
        latency:
          threshold_ms: 1000
      
      # Probabilistic sampling for everything else
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

exporters:
  otlp:
    endpoint: "api.honeycomb.io:443"
    headers:
      "x-honeycomb-team": "${HONEYCOMB_API_KEY}"
      "x-honeycomb-dataset": "${HONEYCOMB_DATASET}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource, tail_sampling]
      exporters: [otlp]
    
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp]

What Gets Sent to Honeycomb

Data Type	Description
Traces	Full request traces with all spans and timing
Spans	Individual operations (DB queries, API calls, etc.)
Events	Structured logs that appear as trace events
Metrics	Request rates, latencies, error counts

Verification

1. Restart Databrain

docker compose restart databrainbackend
# or
kubectl rollout restart deployment/databrain-backend

2. Generate Test Traffic

# Simple health check
curl https://your-databrain-instance.com/api/health

# More complex request
curl -X POST "https://your-databrain-instance.com/api/v2/metric/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"metricId": "test-123"}'

3. View in Honeycomb

1. Go to Honeycomb UI
2. Select your dataset (e.g., databrain-production)
3. You should see traces within seconds (Honeycomb is real-time!)
4. Click on Recent Traces to explore

4. Check Backend Logs

Look for:

{
  "level": "info",
  "message": "[Telemetry] OpenTelemetry initialized - service: databrain-api, endpoint: https://api.honeycomb.io:443"
}

Using Honeycomb Features

1. BubbleUp - Find Anomalies

BubbleUp automatically identifies what makes slow or failing requests different:

1. In Honeycomb, go to your dataset
2. Click New Query
3. Filter: WHERE service.name = databrain-api
4. Visualize: HEATMAP(duration_ms)
5. Click on slow requests cluster
6. Click BubbleUp → Honeycomb shows which attributes correlate with slowness

2. Tracing Waterfall

View full request lifecycle:

1. Click on any trace
2. See the waterfall view with all spans
3. Identify bottlenecks (database queries, external APIs)
4. Click on spans to see attributes

3. High-Cardinality Queries

Query on any attribute combination:

WHERE service.name = databrain-api
  AND http.status_code >= 500
  AND userId EXISTS
GROUP BY userId, http.route
CALCULATE COUNT, P95(duration_ms)

4. Time-based Comparison

Compare performance across time periods:

1. Set time range (e.g., last hour)
2. Click Compare to → Previous period
3. See what changed (requests, latency, errors)

5. Service Map

Visualize service dependencies:

1. Go to Service Map (if enabled)
2. See how databrain-api connects to PostgreSQL, Redis, Hasura, etc.
3. Click on connections to see request rates and error rates

Example Queries

Slowest Endpoints

WHERE service.name = databrain-api
  AND span.kind = server
GROUP BY http.route
CALCULATE P95(duration_ms), COUNT
ORDER BY P95(duration_ms) DESC

Error Analysis

WHERE service.name = databrain-api
  AND status.code = ERROR
GROUP BY error.type, http.route
CALCULATE COUNT

User Experience Analysis

WHERE service.name = databrain-api
  AND userId EXISTS
GROUP BY userId
CALCULATE P95(duration_ms), COUNT
ORDER BY COUNT DESC

Database Query Performance

WHERE service.name = databrain-api
  AND db.system = postgresql
GROUP BY db.statement
CALCULATE P95(duration_ms), COUNT
ORDER BY P95(duration_ms) DESC

Compare Error Rates Over Time

WHERE service.name = databrain-api
GROUP BY status.code
CALCULATE COUNT
COMPARE TO 1 hour ago

Creating Triggers (Alerts)

Set up alerts in Honeycomb:

High Error Rate

1. Go to Triggers → New Trigger
2. Query:

WHERE service.name = databrain-api
  AND status.code = ERROR
CALCULATE COUNT

3. Threshold: Alert when COUNT > 10 per minute
4. Add recipient (email, Slack, PagerDuty)

High Latency

WHERE service.name = databrain-api
  AND span.kind = server
CALCULATE P95(duration_ms)

Threshold: Alert when P95(duration_ms) > 2000

Anomaly Detection

Use Honeycomb’s Anomaly Detection trigger:

• Automatically learns normal behavior
• Alerts on statistical anomalies
• No manual threshold setting required

Boards (Dashboards)

Create custom boards in Honeycomb:

Service Health Board

1. Request Rate

WHERE service.name = databrain-api
CALCULATE COUNT

2. Error Rate

WHERE service.name = databrain-api
CALCULATE COUNT(status.code = ERROR) / COUNT

3. Latency Heatmap

WHERE service.name = databrain-api
HEATMAP(duration_ms)

4. Top Endpoints by Traffic

WHERE service.name = databrain-api
GROUP BY http.route
CALCULATE COUNT

Database Performance Board

1. Query Latency

WHERE db.system = postgresql
HEATMAP(duration_ms)

2. Slowest Queries

WHERE db.system = postgresql
GROUP BY db.statement
CALCULATE P95(duration_ms)
ORDER BY P95(duration_ms) DESC
LIMIT 10

Markers (Deploy Events)

Track deployments with markers:

# After deploying
curl https://api.honeycomb.io/1/markers/databrain-production \
  -X POST \
  -H "X-Honeycomb-Team: ${HONEYCOMB_API_KEY}" \
  -d '{
    "message": "Deployed v1.2.3",
    "type": "deploy",
    "url": "https://github.com/yourorg/databrain/releases/v1.2.3"
  }'

Markers appear as vertical lines in time-series graphs, making it easy to correlate performance changes with deployments.

SLOs (Service Level Objectives)

Create SLOs in Honeycomb:

1. Go to SLOs → New SLO
2. Define SLI (Service Level Indicator):

Example: Availability SLO (99.9%)

Denominator: COUNT (all requests)
WHERE service.name = databrain-api

Numerator: COUNT (successful requests)
WHERE service.name = databrain-api
  AND http.status_code < 500

3. Set target: 99.9% over 30 days
4. Track SLO compliance and error budgets

Troubleshooting

Issue	Solution
No data in Honeycomb	Verify API key is correct and has write permissions
401 Unauthorized	Check x-honeycomb-team header contains valid API key
Data in wrong dataset	Set HONEYCOMB_DATASET explicitly
Missing attributes	Ensure Winston logger includes metadata in logs
High ingestion costs	Implement tail-based sampling in collector

Debug Mode

Enable verbose logging:

LOG_LEVEL=debug
OTEL_LOG_LEVEL=debug

Check logs for export confirmations.

Test API Key

curl https://api.honeycomb.io/1/auth \
  -H "X-Honeycomb-Team: ${HONEYCOMB_API_KEY}"

Should return team information if valid.

Cost Optimization

Honeycomb pricing is based on event volume (spans/events per month).

Free tier: 20M events/month

Optimization strategies:

1. Tail-based sampling: Only send interesting traces (errors, slow requests)
2. Filter health checks: Exclude /health endpoint
3. Sample high-throughput endpoints: Sample 10% of common endpoints
4. Set retention: Adjust in Honeycomb settings (default 60 days)

Sampling Configuration

In collector config:

processors:
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}
      
      - name: sample-baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

This keeps:

• 100% of errors
• 100% of slow requests (>1s)
• 5% of everything else

Best Practices

1. Use Wide Events

Include rich context in your logs:

logger.info('Order processed', {
  orderId: '12345',
  userId: 'user-789',
  amount: 99.99,
  currency: 'USD',
  paymentMethod: 'credit_card',
  processingTime: 245,
  region: 'us-east-1'
});

All these attributes become queryable in Honeycomb.

2. Add Custom Instrumentation

For critical business logic, add custom spans:

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('databrain-business-logic');

async function processOrder(orderId: string) {
  const span = tracer.startSpan('processOrder');
  span.setAttribute('orderId', orderId);
  
  try {
    // Your business logic
    span.addEvent('Order validated');
    
    // ...
    
    span.setStatus({ code: SpanStatusCode.OK });
  } catch (error) {
    span.recordException(error);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw error;
  } finally {
    span.end();
  }
}

3. Use Derived Columns

Create computed columns in Honeycomb:

• Error rate: COUNT_DISTINCT(status.code = ERROR) / COUNT_DISTINCT(trace.id)
• Apdex score: Custom formula for user satisfaction
• Business KPIs: Revenue per request, conversion rate, etc.

Support

• Honeycomb Documentation: https://docs.honeycomb.io/
• Honeycomb Community: https://www.honeycomb.io/community
• Honeycomb Support: Available via in-app chat or email

For Databrain configuration issues, contact your Databrain support team.