- Added a helper function `unwrap_value` to extract values from API responses that may be wrapped in an object format.
- Updated all verification scripts to use `unwrap_value` for fields such as name, description, color, title, and IDs.
- Enhanced checks for boolean fields to handle cases where the API may not return values due to permission issues.
- Ensured consistency across multiple test scripts for better maintainability and readability.
- Implemented verify.sh scripts for the following tests:
- 08-probe: Validate probe resource creation and API response consistency.
- 09-label-crud: Validate label CRUD operations via API.
- 10-monitor-status-crud: Validate monitor status CRUD operations via API.
- 11-incident-severity-crud: Validate incident severity CRUD operations via API.
- 12-status-page-domain: Validate domain and status page resources via API.
- 13-status-page-domain-computed-fields: Validate computed fields for status page domains.
- 14-status-page-server-defaults: Validate server-provided defaults for status pages.
- 15-monitor-server-defaults: Validate server-provided defaults for monitors.
- 16-incident-server-defaults: Validate server-provided defaults for incidents.
- 17-alert-server-defaults: Validate server-provided defaults for alerts.
- 18-scheduled-maintenance-server-defaults: Validate server-provided defaults for scheduled maintenance events.
- 19-on-call-duty-policy-server-defaults: Validate server-provided defaults for on-call duty policies.
Each script checks the API responses against expected values derived from Terraform outputs, ensuring resource integrity and consistency.
- Added a new directory structure for E2E tests under E2E/Terraform/e2e-tests.
- Created scripts for managing the test lifecycle: index.sh, wait-for-services.sh, setup-test-account.sh, run-tests.sh, and cleanup.sh.
- Developed README.md to document the E2E testing process and directory structure.
- Created individual test cases for various resources including labels, monitor statuses, incident severities, incident states, status pages, alert severities, and alert states.
- Configured GitHub Actions workflow for automated testing on pull requests and pushes to main branches.
- Added necessary variable files for each test case to define required inputs.
- Included cookies.txt for session management during API interactions.
- Add Telemetry service entrypoint
- Telemetry/Index.ts: app bootstrap, routes mounting, infrastructure init and Telemetry SDK init.
- Unified queue + worker
- Telemetry/Jobs/TelemetryIngest/ProcessTelemetry.ts: single worker that dispatches queued jobs to specific processors (logs, traces, metrics, syslog, fluent logs).
- Telemetry/Services/Queue/TelemetryQueueService.ts: central queue API and job payload types.
- Per-type Queue wrappers (LogsQueueService, MetricsQueueService, TracesQueueService, FluentLogsQueueService, SyslogQueueService).
- OpenTelemetry ingestion middleware and proto support
- Telemetry/Middleware/OtelRequestMiddleware.ts: detect OTLP endpoint (logs/traces/metrics), decode protobuf bodies using protobufjs and set product type.
- Telemetry/ProtoFiles/OTel/v1/*.proto: include common.proto, logs.proto, metrics.proto, resource.proto, traces.proto for OTLP v1 messages.
- Ingest services
- Telemetry/Services/OtelLogsIngestService.ts: parse incoming OTLP logs, map attributes, convert timestamps, batch insert logs.
- Telemetry/Services/OtelTracesIngestService.ts: parse OTLP traces, build span rows, extract exceptions, batch insert spans and exceptions, save telemetry exception summary.
- Telemetry/Services/OtelMetricsIngestService.ts: parse OTLP metrics, normalize datapoints, batch insert metrics and index metric name -> service map.
- Telemetry/Services/SyslogIngestService.ts: syslog ingestion endpoints, parser integration, map syslog fields to attributes and logs.
- Telemetry/Services/FluentLogsIngestService.ts: ingest Fluentd style logs, normalize entries and insert into log backend.
- Telemetry/Services/OtelIngestBaseService.ts: helpers to resolve service name from attributes/headers.
- Syslog parser and utilities
- Telemetry/Utils/SyslogParser.ts: robust RFC5424 and RFC3164 parser, structured data extraction and sanitization.
- Telemetry/Tests/Utils/SyslogParser.test.ts: unit tests for parser behavior.
- Telemetry exception utilities
- Telemetry/Utils/Exception.ts: generate exception fingerprint and upsert telemetry exception status (saveOrUpdateTelemetryException).
- Queue & job integration
- New integration with Common/Server/Infrastructure/Queue and QueueWorker, job id generation and telemetry job types.
- Telemetry services add ingestion jobs instead of processing synchronously.
- Config, build and dev tooling
- Add Telemetry/package.json, package-lock.json, tsconfig.json, nodemon.json, jest config.
- New script configs and dependencies (protobufjs, ts-node, jest, nodemon, etc).
- Docker / environment updates
- docker-compose.base.yml, docker-compose.dev.yml, docker-compose.yml: rename service from open-telemetry-ingest -> telemetry and wire TELEMETRY_* envs.
- config.example.env: rename and consolidate environment variables (OPEN_TELEMETRY_* -> TELEMETRY_*, update hostnames and ports).
- Tests/Scripts/status-check.sh: update ready-check target to telemetry/status/ready.
- Other
- Telemetry/Services/Queue/*: export helpers and legacy-compatible job interface shims.
- Memory cleanup and batching safeguards across ingest services.
- Logging and capture spans added to key code paths.
BREAKING CHANGES / MIGRATION NOTES:
- Environment variables and docker service names changed:
- Replace OPEN_TELEMETRY_... vars with TELEMETRY_... (PORT, HOSTNAME, CONCURRENCY, DISABLE_TELEMETRY, etc).
- docker-compose entries moved from "open-telemetry-ingest" to "telemetry" and image name changed to oneuptime/telemetry.
- Update any deployment automation and monitoring checks referencing the old service name or endpoints.
- Consumers: OTLP endpoints and behavior remain supported, but ingestion is now queued and processed asynchronously.
Testing / Running:
- Install deps in Telemetry/ (npm install) after syncing Common workspace.
- Run dev: npx nodemon (nodemon.json) or build & start using provided scripts.
- Run tests with jest (Telemetry test suite includes SyslogParser unit tests).
Files added/modified (high level):
- Added many files under Telemetry/: Index, Jobs, Middleware, ProtoFiles, Services, Utils, Tests, package and config artifacts.
- Modified docker-compose.* and config.example.env and status check script to use new TELEMETRY service/vars.
- Update lockfiles for root, TestServer, Worker and Workflow
- Add new/top-level packages and types:
- @simplewebauthn/server, archiver, botbuilder
- @types/archiver
- Bump notable package versions:
- axios, nodemailer, playwright, react-syntax-highlighter
- Sync a large set of transitive dependency upgrades:
- @babel/* (code-frame, parser, template, types, helpers, helper-*), brace-expansion, braces, fill-range, micromatch, picocolors, cross-spawn, to-regex-range and others
- Remove/clean outdated lock entries (e.g. to-fast-properties removed where unnecessary)
- Add/fix license fields for several packages in the lockfiles
- General lockfile normalization to ensure consistent subdependency versions across packages
This keeps dependency trees aligned across monorepo packages and pulls in recent bug/security fixes from transitive upgrades.
- Updated comments in Probe/Config.ts to use block comments for proxy configuration.
- Refactored comments in PortMonitor.ts, SyntheticMonitor.ts, and OnlineCheck.ts to block comments for better readability.
- Adjusted comments in ProbeIngest/API/Monitor.ts and ProbeIngest/API/Probe.ts to block comments for clarity.
- Standardized comments in various data migration scripts to block comments for consistency.
- Modified eslint.config.js to enforce multiline comment style as an error.
