diff --git a/Internal/Roadmap/Logs.md b/Internal/Roadmap/Logs.md index 4f643282a9..adb2c66e49 100644 --- a/Internal/Roadmap/Logs.md +++ b/Internal/Roadmap/Logs.md @@ -19,6 +19,7 @@ The following features have been implemented and removed from this plan: - **Phase 2.1** - Saved Views (LogSavedView model, SavedViewsDropdown, CRUD API) - **Phase 2.2** - Log Analytics View (LogsAnalyticsView with timeseries, toplist, table charts; analytics endpoint) - **Phase 2.3** - Column Customization (ColumnSelector with dynamic columns from log attributes) +- **Phase 5.8** - Store Missing OpenTelemetry Log Fields (observedTimeUnixNano, droppedAttributesCount, flags columns + ingestion + migration) ## Gap Analysis Summary @@ -197,42 +198,17 @@ These optimizations address fundamental storage and indexing gaps in the telemet - `Common/Server/Utils/AnalyticsDatabase/StatementGenerator.ts` (emit PROJECTION clause) - `Worker/DataMigrations/` (new migration to materialize) -### 5.8 Store Missing OpenTelemetry Log Fields (Low) - -**Current**: Several standard OTEL log record fields are dropped during ingestion: -- `observedTimeUnixNano` — when the log was collected by the pipeline (useful for measuring ingestion lag) -- `droppedAttributesCount` — signals data loss during collection -- `flags` — log record flags (e.g., W3C trace flags) - -**Target**: Preserve these fields for full OTEL compliance and operational debugging. - -**Implementation**: - -- Add optional columns to the Log model: - - `observedTimeUnixNano` (LongNumber) - - `droppedAttributesCount` (Number, default 0) - - `flags` (Number, default 0) -- Update `OtelLogsIngestService.processLogsAsync()` to extract and store these fields -- Migration to add columns to existing table - -**Files to modify**: -- `Common/Models/AnalyticsModels/Log.ts` (add columns) -- `Telemetry/Services/OtelLogsIngestService.ts` (extract additional fields) -- `Worker/DataMigrations/` (new migration) - ### 5.x Remaining Performance Impact Summary | Optimization | Query Pattern Improved | Expected Speedup | Effort | |-------------|----------------------|-------------------|--------| | 5.3 DateTime64 time column | Sub-second log ordering | Correctness fix | Medium | | 5.7 Histogram projections | Histogram and severity aggregation | 5-10x | Medium | -| 5.8 Missing OTEL fields | OTEL compliance | N/A (completeness) | Small | ### 5.x Recommended Remaining Order 1. **5.3** — DateTime64 upgrade (correctness) 2. **5.7** — Projections (performance polish) -3. **5.8** — Missing OTEL fields (completeness) ---