feat(Logs): add missing OpenTelemetry log fields to enhance compliance and debugging

This commit is contained in:
Nawaz Dhandala
2026-03-12 21:38:58 +00:00
parent 03c38be19c
commit d394c96eb9

View File

@@ -19,6 +19,7 @@ The following features have been implemented and removed from this plan:
- **Phase 2.1** - Saved Views (LogSavedView model, SavedViewsDropdown, CRUD API)
- **Phase 2.2** - Log Analytics View (LogsAnalyticsView with timeseries, toplist, table charts; analytics endpoint)
- **Phase 2.3** - Column Customization (ColumnSelector with dynamic columns from log attributes)
- **Phase 5.8** - Store Missing OpenTelemetry Log Fields (observedTimeUnixNano, droppedAttributesCount, flags columns + ingestion + migration)
## Gap Analysis Summary
@@ -197,42 +198,17 @@ These optimizations address fundamental storage and indexing gaps in the telemet
- `Common/Server/Utils/AnalyticsDatabase/StatementGenerator.ts` (emit PROJECTION clause)
- `Worker/DataMigrations/` (new migration to materialize)
### 5.8 Store Missing OpenTelemetry Log Fields (Low)
**Current**: Several standard OTEL log record fields are dropped during ingestion:
- `observedTimeUnixNano` — when the log was collected by the pipeline (useful for measuring ingestion lag)
- `droppedAttributesCount` — signals data loss during collection
- `flags` — log record flags (e.g., W3C trace flags)
**Target**: Preserve these fields for full OTEL compliance and operational debugging.
**Implementation**:
- Add optional columns to the Log model:
- `observedTimeUnixNano` (LongNumber)
- `droppedAttributesCount` (Number, default 0)
- `flags` (Number, default 0)
- Update `OtelLogsIngestService.processLogsAsync()` to extract and store these fields
- Migration to add columns to existing table
**Files to modify**:
- `Common/Models/AnalyticsModels/Log.ts` (add columns)
- `Telemetry/Services/OtelLogsIngestService.ts` (extract additional fields)
- `Worker/DataMigrations/` (new migration)
### 5.x Remaining Performance Impact Summary
| Optimization | Query Pattern Improved | Expected Speedup | Effort |
|-------------|----------------------|-------------------|--------|
| 5.3 DateTime64 time column | Sub-second log ordering | Correctness fix | Medium |
| 5.7 Histogram projections | Histogram and severity aggregation | 5-10x | Medium |
| 5.8 Missing OTEL fields | OTEL compliance | N/A (completeness) | Small |
### 5.x Recommended Remaining Order
1. **5.3** — DateTime64 upgrade (correctness)
2. **5.7** — Projections (performance polish)
3. **5.8** — Missing OTEL fields (completeness)
---