github/oneuptime
1
1
Fork 0
mirror of https://github.com/OneUptime/oneuptime.git synced 2026-04-06 00:32:12 +02:00
Code Issues 193 Packages Projects Releases Wiki Activity

chore(fluent-ingest): migrate fluent log ingest into open-telemetry-ingest and remove legacy fluent-ingest service

Browse Source 
- Move Fluent/Fluent Bit logs ingestion into open-telemetry-ingest:
  - Add OpenTelemetryIngest/API/Fluent.ts (routes for /fluentd and queue endpoints)
  - Add Queue service, job worker and processor:
    - OpenTelemetryIngest/Services/Queue/FluentLogsQueueService.ts
    - OpenTelemetryIngest/Jobs/TelemetryIngest/ProcessFluentLogs.ts
  - Register Fluent API and job processing in OpenTelemetryIngest/Index.ts
  - Introduce QueueName.FluentLogs and related queue usage

- Remove legacy FluentIngest service and configuration:
  - Delete fluent-ingest docker-compose/dev/base entries and docker-compose.yml service
  - Remove fluent-ingest related helm values, KEDA scaledobject, ingress host and schema entries
  - Remove FLUENTD_HOST env/values and replace FLUENT_INGEST_HOSTNAME -> FLUENT_LOGS_HOSTNAME (pointing to open-telemetry-ingest)
  - Update config.example.env keys (FLUENT_LOGS_CONCURRENCY, DISABLE_TELEMETRY_FOR_FLUENT_LOGS)
  - Remove FluentIngestRoute and FLUENT_INGEST_URL/hostname usages from UI config/templates
  - Remove VSCode launch debug config for Fluent Ingest
  - Remove Fluent ingest E2E status check entry in Tests/Scripts/status-check.sh
  - Update docs/architecture diagram and Helm templates to reflect "FluentLogs" / Fluent Bit flow

- Misc:
  - Remove FLUENTD_HOST environment injection from docker-compose.base.yml
  - Cleanup related values.schema.json and values.yaml entries

This consolidates log ingestion under the OpenTelemetry ingest service and removes the separate FluentIngest service and its configuration.
This commit is contained in:
Simon Larsen
2025-11-07 19:37:31 +00:00
parent b31d1076b8
commit a80b7ba88c
19 changed files with 381 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
config.example.env
View File

@@ -59,8 +59,6 @@ COMPOSE_PROJECT_NAME=oneuptime
# OTEL HOST - if you like the collector to be hosted on a different server then change this to the IP of the server.
OTEL_COLLECTOR_HOST=
# FLUENTD_HOST - if you like the fluentd to be hosted on a different server then change this to the IP of the server.
FLUENTD_HOST=
# Clickhouse Settings
CLICKHOUSE_USER=default
@@ -94,7 +92,7 @@ REDIS_TLS_SENTINEL_MODE=false
# Hostnames. Usually does not need to change.
PROBE_INGEST_HOSTNAME=probe-ingest:3400
FLUENT_INGEST_HOSTNAME=fluent-ingest:3401
FLUENT_LOGS_HOSTNAME=open-telemetry-ingest:3403
INCOMING_REQUEST_INGEST_HOSTNAME=incoming-request-ingest:3402
OPEN_TELEMETRY_INGEST_HOSTNAME=otel-telemetry-ingest:3403
@@ -248,8 +246,8 @@ WORKFLOW_TIMEOUT_IN_MS=5000
# Max number of telemetry jobs processed concurrently by OpenTelemetry Ingest worker
OPEN_TELEMETRY_INGEST_CONCURRENCY=100
# Max number of jobs processed concurrently by Fluent Ingest worker
FLUENT_INGEST_CONCURRENCY=100
# Max number of jobs processed concurrently by Fluent Logs worker
FLUENT_LOGS_CONCURRENCY=100
# Max number of jobs processed concurrently by Incoming Request Ingest worker
INCOMING_REQUEST_INGEST_CONCURRENCY=100
@@ -316,7 +314,7 @@ DISABLE_TELEMETRY_FOR_ACCOUNTS=true
DISABLE_TELEMETRY_FOR_APP=true
DISABLE_TELEMETRY_FOR_PROBE_INGEST=true
DISABLE_TELEMETRY_FOR_OPEN_TELEMETRY_INGEST=true
DISABLE_TELEMETRY_FOR_FLUENT_INGEST=true
DISABLE_TELEMETRY_FOR_FLUENT_LOGS=true
DISABLE_TELEMETRY_FOR_INCOMING_REQUEST_INGEST=true
DISABLE_TELEMETRY_FOR_TEST_SERVER=true
DISABLE_TELEMETRY_FOR_STATUS_PAGE=true