feat(telemetry): add new Telemetry service (OTel, Syslog, Fluent, Metrics, Traces) and unified ingestion pipeline

- Add Telemetry service entrypoint
  - Telemetry/Index.ts: app bootstrap, routes mounting, infrastructure init and Telemetry SDK init.

- Unified queue + worker
  - Telemetry/Jobs/TelemetryIngest/ProcessTelemetry.ts: single worker that dispatches queued jobs to specific processors (logs, traces, metrics, syslog, fluent logs).
  - Telemetry/Services/Queue/TelemetryQueueService.ts: central queue API and job payload types.
  - Per-type Queue wrappers (LogsQueueService, MetricsQueueService, TracesQueueService, FluentLogsQueueService, SyslogQueueService).

- OpenTelemetry ingestion middleware and proto support
  - Telemetry/Middleware/OtelRequestMiddleware.ts: detect OTLP endpoint (logs/traces/metrics), decode protobuf bodies using protobufjs and set product type.
  - Telemetry/ProtoFiles/OTel/v1/*.proto: include common.proto, logs.proto, metrics.proto, resource.proto, traces.proto for OTLP v1 messages.

- Ingest services
  - Telemetry/Services/OtelLogsIngestService.ts: parse incoming OTLP logs, map attributes, convert timestamps, batch insert logs.
  - Telemetry/Services/OtelTracesIngestService.ts: parse OTLP traces, build span rows, extract exceptions, batch insert spans and exceptions, save telemetry exception summary.
  - Telemetry/Services/OtelMetricsIngestService.ts: parse OTLP metrics, normalize datapoints, batch insert metrics and index metric name -> service map.
  - Telemetry/Services/SyslogIngestService.ts: syslog ingestion endpoints, parser integration, map syslog fields to attributes and logs.
  - Telemetry/Services/FluentLogsIngestService.ts: ingest Fluentd style logs, normalize entries and insert into log backend.
  - Telemetry/Services/OtelIngestBaseService.ts: helpers to resolve service name from attributes/headers.

- Syslog parser and utilities
  - Telemetry/Utils/SyslogParser.ts: robust RFC5424 and RFC3164 parser, structured data extraction and sanitization.
  - Telemetry/Tests/Utils/SyslogParser.test.ts: unit tests for parser behavior.

- Telemetry exception utilities
  - Telemetry/Utils/Exception.ts: generate exception fingerprint and upsert telemetry exception status (saveOrUpdateTelemetryException).

- Queue & job integration
  - New integration with Common/Server/Infrastructure/Queue and QueueWorker, job id generation and telemetry job types.
  - Telemetry services add ingestion jobs instead of processing synchronously.

- Config, build and dev tooling
  - Add Telemetry/package.json, package-lock.json, tsconfig.json, nodemon.json, jest config.
  - New script configs and dependencies (protobufjs, ts-node, jest, nodemon, etc).

- Docker / environment updates
  - docker-compose.base.yml, docker-compose.dev.yml, docker-compose.yml: rename service from open-telemetry-ingest -> telemetry and wire TELEMETRY_* envs.
  - config.example.env: rename and consolidate environment variables (OPEN_TELEMETRY_* -> TELEMETRY_*, update hostnames and ports).
  - Tests/Scripts/status-check.sh: update ready-check target to telemetry/status/ready.

- Other
  - Telemetry/Services/Queue/*: export helpers and legacy-compatible job interface shims.
  - Memory cleanup and batching safeguards across ingest services.
  - Logging and capture spans added to key code paths.

BREAKING CHANGES / MIGRATION NOTES:
- Environment variables and docker service names changed:
  - Replace OPEN_TELEMETRY_... vars with TELEMETRY_... (PORT, HOSTNAME, CONCURRENCY, DISABLE_TELEMETRY, etc).
  - docker-compose entries moved from "open-telemetry-ingest" to "telemetry" and image name changed to oneuptime/telemetry.
  - Update any deployment automation and monitoring checks referencing the old service name or endpoints.
- Consumers: OTLP endpoints and behavior remain supported, but ingestion is now queued and processed asynchronously.

Testing / Running:
- Install deps in Telemetry/ (npm install) after syncing Common workspace.
- Run dev: npx nodemon (nodemon.json) or build & start using provided scripts.
- Run tests with jest (Telemetry test suite includes SyslogParser unit tests).

Files added/modified (high level):
- Added many files under Telemetry/: Index, Jobs, Middleware, ProtoFiles, Services, Utils, Tests, package and config artifacts.
- Modified docker-compose.* and config.example.env and status check script to use new TELEMETRY service/vars.
This commit is contained in:
Nawaz Dhandala
2025-11-07 21:36:47 +00:00
parent 351fc4828b
commit f49b1995df
63 changed files with 204 additions and 209 deletions

View File

@@ -92,15 +92,14 @@ REDIS_TLS_SENTINEL_MODE=false
# Hostnames. Usually does not need to change.
PROBE_INGEST_HOSTNAME=probe-ingest:3400
FLUENT_LOGS_HOSTNAME=open-telemetry-ingest:3403
INCOMING_REQUEST_INGEST_HOSTNAME=incoming-request-ingest:3402
OPEN_TELEMETRY_INGEST_HOSTNAME=otel-telemetry-ingest:3403
TELEMETRY_HOSTNAME=telemetry:3403
SERVER_ACCOUNTS_HOSTNAME=accounts
SERVER_APP_HOSTNAME=app
SERVER_PROBE_INGEST_HOSTNAME=probe-ingest
SERVER_SERVER_MONITOR_INGEST_HOSTNAME=server-monitor-ingest
SERVER_OPEN_TELEMETRY_INGEST_HOSTNAME=otel-telemetry-ingest
SERVER_TELEMETRY_HOSTNAME=telemetry
SERVER_INCOMING_REQUEST_INGEST_HOSTNAME=incoming-request-ingest
SERVER_TEST_SERVER_HOSTNAME=test-server
SERVER_STATUS_PAGE_HOSTNAME=status-page
@@ -116,7 +115,7 @@ SERVER_DOCS_HOSTNAME=docs
APP_PORT=3002
PROBE_INGEST_PORT=3400
SERVER_MONITOR_INGEST_PORT=3404
OPEN_TELEMETRY_INGEST_PORT=3403
TELEMETRY_PORT=3403
INCOMING_REQUEST_INGEST_PORT=3402
TEST_SERVER_PORT=3800
ACCOUNTS_PORT=3003
@@ -244,7 +243,7 @@ WORKFLOW_TIMEOUT_IN_MS=5000
# Concurrency settings
# Max number of telemetry jobs processed concurrently by OpenTelemetry Ingest worker
OPEN_TELEMETRY_INGEST_CONCURRENCY=100
TELEMETRY_CONCURRENCY=100
# Max number of jobs processed concurrently by Fluent Logs worker
FLUENT_LOGS_CONCURRENCY=100
@@ -313,7 +312,7 @@ LLM_SERVER_HUGGINGFACE_MODEL_NAME=
DISABLE_TELEMETRY_FOR_ACCOUNTS=true
DISABLE_TELEMETRY_FOR_APP=true
DISABLE_TELEMETRY_FOR_PROBE_INGEST=true
DISABLE_TELEMETRY_FOR_OPEN_TELEMETRY_INGEST=true
DISABLE_TELEMETRY_FOR_TELEMETRY=true
DISABLE_TELEMETRY_FOR_FLUENT_LOGS=true
DISABLE_TELEMETRY_FOR_INCOMING_REQUEST_INGEST=true
DISABLE_TELEMETRY_FOR_TEST_SERVER=true