From a94e48501e2d28166ffd09d42b78da45f54814b4 Mon Sep 17 00:00:00 2001 From: Nawaz Dhandala Date: Thu, 12 Mar 2026 21:23:47 +0000 Subject: [PATCH] feat(Logs): update gap analysis and implementation phases for log management features --- Internal/Roadmap/Logs.md | 81 ++++------------------------------------ 1 file changed, 8 insertions(+), 73 deletions(-) diff --git a/Internal/Roadmap/Logs.md b/Internal/Roadmap/Logs.md index 9008c62c0a..4f643282a9 100644 --- a/Internal/Roadmap/Logs.md +++ b/Internal/Roadmap/Logs.md @@ -16,15 +16,15 @@ The following features have been implemented and removed from this plan: - **Phase 5.4** - Per-service TTL via `retentionDate` column - **Phase 5.5** - Parameterized SQL in `LogAggregationService` - **Phase 5.6** - ZSTD compression on `body` column +- **Phase 2.1** - Saved Views (LogSavedView model, SavedViewsDropdown, CRUD API) +- **Phase 2.2** - Log Analytics View (LogsAnalyticsView with timeseries, toplist, table charts; analytics endpoint) +- **Phase 2.3** - Column Customization (ColumnSelector with dynamic columns from log attributes) ## Gap Analysis Summary | Feature | OneUptime | Datadog | New Relic | Priority | |---------|-----------|---------|-----------|----------| -| Saved Views | None | Full state save/share | Full state save/share | **P1** | | Log Patterns (ML clustering) | None | Auto-clustering + Pattern Inspector | ML clustering + anomaly | **P1** | -| Log-based analytics/charts | None | Timeseries, TopList, Table, Pie | Full NRQL charting | **P1** | -| Column customization | Fixed 4 columns | Fully customizable | Configurable | **P1** | | Log context (surrounding logs) | None | Before/after from same host/service | Automatic via APM agent | **P2** | | Log Pipelines (server-side processing) | None (raw storage only) | 270+ OOTB, 14+ processor types | Grok parsing, built-in rules | **P2** | | Log-based Metrics | None | Count + Distribution, 15-month retention | Via NRQL | **P2** | @@ -36,68 +36,6 @@ The following features have been implemented and removed from this plan: --- -## Phase 2: Analytics & Organization (P1) — Power User Features - -### 2.1 Saved Views - -**Current**: No way to save filter/query state. -**Target**: Users can save, name, and share log views. - -**Implementation**: - -- Create a new PostgreSQL model `LogSavedView` with fields: `id`, `projectId`, `name`, `query` (JSON), `columns` (JSON array), `sortField`, `sortOrder`, `pageSize`, `createdByUserId`, `isDefault`, timestamps -- CRUD API via standard OneUptime model patterns -- Add a "Save View" button in the toolbar and a dropdown to load saved views -- Saved views appear in the left sidebar above the facets - -**Files to modify**: -- `Common/Models/DatabaseModels/LogSavedView.ts` (new model) -- `Common/Server/Services/LogSavedViewService.ts` (new service) -- `Common/UI/Components/LogsViewer/components/SavedViewsDropdown.tsx` (new) -- `Common/UI/Components/LogsViewer/components/LogsViewerToolbar.tsx` (add save/load buttons) - -### 2.2 Log Analytics View (Charts from Logs) - -**Current**: Logs are only viewable as a list. -**Target**: A toggle to switch from "List" to "Analytics" mode showing aggregate visualizations. - -**Implementation**: - -- Add a view mode toggle in the toolbar: "List" | "Analytics" -- Analytics view provides a query builder for: - - **Timeseries**: Count/unique count over time, grouped by up to 2 dimensions - - **Top List**: Top N values for a dimension by count - - **Table**: Pivot table with multiple group-by dimensions -- Reuse the histogram endpoint with extended aggregation support -- New API endpoint `POST /telemetry/logs/analytics` that supports flexible GROUP BY + aggregation queries -- Create `LogsAnalyticsView` component using a charting library (recommend recharts, already likely in the project) - -**Files to modify**: -- `Common/Server/API/TelemetryAPI.ts` (add analytics endpoint) -- `Common/UI/Components/LogsViewer/components/LogsAnalyticsView.tsx` (new) -- `Common/UI/Components/LogsViewer/components/LogsViewerToolbar.tsx` (add view toggle) -- `Common/UI/Components/LogsViewer/LogsViewer.tsx` (conditional rendering) - -### 2.3 Column Customization - -**Current**: Fixed columns: Time, Service, Severity, Message. -**Target**: Users can add/remove/reorder columns from log attributes. - -**Implementation**: - -- Add a "Columns" button in the toolbar that opens a dropdown/popover -- Default columns: Time, Service, Severity, Message -- Available columns: any discovered attribute key (from the existing get-attributes endpoint) -- Selected columns persist in localStorage (and in Saved Views when that feature ships) -- Extend `LogsTable.tsx` to dynamically render columns based on configuration - -**Files to modify**: -- `Common/UI/Components/LogsViewer/components/ColumnSelector.tsx` (new) -- `Common/UI/Components/LogsViewer/components/LogsTable.tsx` (dynamic columns) -- `Common/UI/Components/LogsViewer/LogsViewer.tsx` (column state management) - ---- - ## Phase 3: Processing & Operations (P2) — Platform Capabilities ### 3.1 Log Context (Surrounding Logs) @@ -195,14 +133,11 @@ The following features have been implemented and removed from this plan: ## Recommended Implementation Order -1. **Phase 2.3** - Column Customization (small effort, high user value) -2. **Phase 2.1** - Saved Views (moderate effort, high retention value) -3. **Phase 3.4** - Export CSV/JSON (small effort, table-stakes feature) -4. **Phase 3.1** - Log Context (moderate effort, high debugging value) -5. **Phase 2.2** - Log Analytics View (larger effort, advanced user feature) -6. **Phase 3.2** - Log Pipelines (large effort, platform capability) -7. **Phase 3.3** - Drop Filters (moderate effort, cost optimization) -8. **Phase 4.x** - Patterns, Shortcuts, Data Scrubbing (future) +1. **Phase 3.4** - Export CSV/JSON (small effort, table-stakes feature) +2. **Phase 3.1** - Log Context (moderate effort, high debugging value) +3. **Phase 3.2** - Log Pipelines (large effort, platform capability) +4. **Phase 3.3** - Drop Filters (moderate effort, cost optimization) +5. **Phase 4.x** - Patterns, Shortcuts, Data Scrubbing (future) ## Phase 5: ClickHouse Storage & Query Optimizations (P0) — Performance Foundation