- Added a new directory structure for E2E tests under E2E/Terraform/e2e-tests.
- Created scripts for managing the test lifecycle: index.sh, wait-for-services.sh, setup-test-account.sh, run-tests.sh, and cleanup.sh.
- Developed README.md to document the E2E testing process and directory structure.
- Created individual test cases for various resources including labels, monitor statuses, incident severities, incident states, status pages, alert severities, and alert states.
- Configured GitHub Actions workflow for automated testing on pull requests and pushes to main branches.
- Added necessary variable files for each test case to define required inputs.
- Included cookies.txt for session management during API interactions.
- Deleted RunCommandTool, SearchWorkspaceTool, WriteFileTool, and their associated interfaces and implementations.
- Removed Tool, ToolRegistry, and AgentLogger classes, along with their dependencies.
- Eliminated utility functions for workspace path management and secret sanitization.
- Cleaned up TypeScript configuration and example environment variables related to Copilot.
- Updated Docker Compose files to remove references to Copilot services.
- Updated versioning in release.yml to use only major_minor version for Helm chart packaging, Docker images, and GitHub releases.
- Adjusted versioning in test-release.yaml to reflect similar changes, removing build number suffix for test releases.
- Enhanced the `test-release.yaml` workflow to read and determine semantic versioning from `VERSION_PREFIX`, including major, minor, and patch components.
- Adjusted versioning format in the workflow to use a new scheme: `major.minor.patch-test+build.build_number`.
- Updated Docker image build script to sanitize version strings by replacing '+' with '-' for tagging.
- Incremented the version in `VERSION_PREFIX` from `9.0` to `9.1.0`.
- Implemented a bash script to build and push Docker images.
- Added support for multiple platforms using docker buildx.
- Included options for specifying image name, version, Dockerfile path, build context, and additional tags.
- Integrated Git SHA detection for build arguments.
- Provided usage instructions and error handling for missing required arguments.
- Add Telemetry service entrypoint
- Telemetry/Index.ts: app bootstrap, routes mounting, infrastructure init and Telemetry SDK init.
- Unified queue + worker
- Telemetry/Jobs/TelemetryIngest/ProcessTelemetry.ts: single worker that dispatches queued jobs to specific processors (logs, traces, metrics, syslog, fluent logs).
- Telemetry/Services/Queue/TelemetryQueueService.ts: central queue API and job payload types.
- Per-type Queue wrappers (LogsQueueService, MetricsQueueService, TracesQueueService, FluentLogsQueueService, SyslogQueueService).
- OpenTelemetry ingestion middleware and proto support
- Telemetry/Middleware/OtelRequestMiddleware.ts: detect OTLP endpoint (logs/traces/metrics), decode protobuf bodies using protobufjs and set product type.
- Telemetry/ProtoFiles/OTel/v1/*.proto: include common.proto, logs.proto, metrics.proto, resource.proto, traces.proto for OTLP v1 messages.
- Ingest services
- Telemetry/Services/OtelLogsIngestService.ts: parse incoming OTLP logs, map attributes, convert timestamps, batch insert logs.
- Telemetry/Services/OtelTracesIngestService.ts: parse OTLP traces, build span rows, extract exceptions, batch insert spans and exceptions, save telemetry exception summary.
- Telemetry/Services/OtelMetricsIngestService.ts: parse OTLP metrics, normalize datapoints, batch insert metrics and index metric name -> service map.
- Telemetry/Services/SyslogIngestService.ts: syslog ingestion endpoints, parser integration, map syslog fields to attributes and logs.
- Telemetry/Services/FluentLogsIngestService.ts: ingest Fluentd style logs, normalize entries and insert into log backend.
- Telemetry/Services/OtelIngestBaseService.ts: helpers to resolve service name from attributes/headers.
- Syslog parser and utilities
- Telemetry/Utils/SyslogParser.ts: robust RFC5424 and RFC3164 parser, structured data extraction and sanitization.
- Telemetry/Tests/Utils/SyslogParser.test.ts: unit tests for parser behavior.
- Telemetry exception utilities
- Telemetry/Utils/Exception.ts: generate exception fingerprint and upsert telemetry exception status (saveOrUpdateTelemetryException).
- Queue & job integration
- New integration with Common/Server/Infrastructure/Queue and QueueWorker, job id generation and telemetry job types.
- Telemetry services add ingestion jobs instead of processing synchronously.
- Config, build and dev tooling
- Add Telemetry/package.json, package-lock.json, tsconfig.json, nodemon.json, jest config.
- New script configs and dependencies (protobufjs, ts-node, jest, nodemon, etc).
- Docker / environment updates
- docker-compose.base.yml, docker-compose.dev.yml, docker-compose.yml: rename service from open-telemetry-ingest -> telemetry and wire TELEMETRY_* envs.
- config.example.env: rename and consolidate environment variables (OPEN_TELEMETRY_* -> TELEMETRY_*, update hostnames and ports).
- Tests/Scripts/status-check.sh: update ready-check target to telemetry/status/ready.
- Other
- Telemetry/Services/Queue/*: export helpers and legacy-compatible job interface shims.
- Memory cleanup and batching safeguards across ingest services.
- Logging and capture spans added to key code paths.
BREAKING CHANGES / MIGRATION NOTES:
- Environment variables and docker service names changed:
- Replace OPEN_TELEMETRY_... vars with TELEMETRY_... (PORT, HOSTNAME, CONCURRENCY, DISABLE_TELEMETRY, etc).
- docker-compose entries moved from "open-telemetry-ingest" to "telemetry" and image name changed to oneuptime/telemetry.
- Update any deployment automation and monitoring checks referencing the old service name or endpoints.
- Consumers: OTLP endpoints and behavior remain supported, but ingestion is now queued and processed asynchronously.
Testing / Running:
- Install deps in Telemetry/ (npm install) after syncing Common workspace.
- Run dev: npx nodemon (nodemon.json) or build & start using provided scripts.
- Run tests with jest (Telemetry test suite includes SyslogParser unit tests).
Files added/modified (high level):
- Added many files under Telemetry/: Index, Jobs, Middleware, ProtoFiles, Services, Utils, Tests, package and config artifacts.
- Modified docker-compose.* and config.example.env and status check script to use new TELEMETRY service/vars.
- Remove FluentIngest CI jobs and release/test deploy steps (build.yml, compile.yml, release.yml, test-release.yaml)
- Delete test.fluent-ingest workflow
- Remove Fluent-related env vars/hostnames from config.example.env, docker-compose.base.yml, HelmChart templates and EnvironmentConfig.ts
- Remove fluent-ingest service block from docker-compose.base.yml
- Bump Playwright and TypeORM versions in OpenTelemetryIngest package-lock.json
Add --no-cache to docker build and docker buildx commands across build.yml, release.yml and test-release.yaml to force fresh image builds and avoid using cached layers.
