# Plan: Bring OneUptime Metrics to Industry Parity and Beyond

## Context

OneUptime's metrics implementation provides OTLP ingestion (HTTP and gRPC), ClickHouse storage with support for Gauge, Sum, Histogram, and ExponentialHistogram metric types, basic aggregations (Avg, Sum, Min, Max, Count), single-attribute GROUP BY, formula support for calculated metrics, threshold-based metric monitors, and a metric explorer with line/bar charts. Auto-discovery creates MetricType metadata (name, description, unit) on first ingest. Per-service data retention with TTL (default 15 days).

This plan identifies the remaining gaps vs DataDog and New Relic, and proposes a phased implementation to close them and build a best-in-class metrics product.

## Completed

The following features have been implemented:
- **OTLP Ingestion** - HTTP and gRPC metric ingestion with async queue-based batch processing
- **Metric Types** - Gauge, Sum, Histogram, ExponentialHistogram support
- **ClickHouse Storage** - MergeTree with `sipHash64(projectId) % 16` partitioning, per-service TTL
- **Aggregations** - Avg, Sum, Min, Max, Count
- **Single-Attribute GROUP BY** - Group by one attribute at a time
- **Formulas** - Calculated metrics using aliases (e.g., `a / b * 100`)
- **Metric Explorer** - Time range selection, multiple queries with aliases, URL state persistence
- **Threshold-Based Monitors** - Static threshold alerting on aggregated metric values
- **MetricType Auto-Discovery** - Name, description, unit captured on first ingest
- **Attribute Storage** - Full JSON with extracted `attributeKeys` array for fast enumeration
- **BloomFilter index** on `name`, Set index on `serviceType`
- **Phase 2.3** - Render Metric Units on Charts (MetricType.unit passed through to chart Y-axis and tooltip formatting)
- **S.1** - Fix Sort Key Order (sort key changed to `projectId, name, serviceId, time` for optimal metric name filtering)
- **S.2** - Upgrade time to DateTime64 (time column uses `TableColumnType.DateTime64` for sub-second precision)
- **S.3** - Add Skip Index on metricPointType (Set skip index `idx_metric_point_type` added to Metric model)

## Gap Analysis Summary

| Feature | OneUptime | DataDog | New Relic | Priority |
|---------|-----------|---------|-----------|----------|
| Percentile aggregations (p50/p75/p90/p95/p99) | None | DDSketch distributions | NRQL percentile() | **P0** |
| Rate/derivative calculations | None | Native Rate type + .as_rate() | rate() NRQL function | **P0** |
| Multi-attribute GROUP BY | Single attribute only | Multiple tags | FACET on multiple attrs | **P0** |
| Rollup/downsampling for long-range queries | None (raw data, 15-day TTL) | Automatic tiered rollups | 30-day raw + 13-month rollups | **P0** |
| Anomaly detection | Static thresholds only | Watchdog + anomaly monitors | Anomaly detection + sigma bands | **P1** |
| SLO/SLI tracking | None | Metric-based + Time Slice SLOs | One-click setup + error budgets | **P1** |
| Heatmap visualization | None | Purpose-built for distributions | Built-in chart type | **P1** |
| Time-over-time comparison | None | Yes | COMPARE WITH in NRQL | **P1** |
| Summary metric type | Not supported | N/A (uses Distribution) | Yes | **P1** |
| Query language | Form-based UI only | Graphing editor + NLQ | Full NRQL language | **P2** |
| Predictive alerting | None | Watchdog forecasting | GA predictive alerting | **P2** |
| Metric correlations | None | Auto-surfaces related metrics | Applied Intelligence correlation | **P2** |
| Golden Signals dashboards | None | Available via APM | Pre-built with default alerts | **P2** |
| Cardinality management | None | Metrics Without Limits + Explorer | Budget system + pruning rules | **P2** |
| More chart types | Line and bar only | 12+ types | 10+ types with conditional coloring | **P2** |
| Dashboard templates | None | Pre-built integration dashboards | Pre-built entity dashboards | **P2** |
| Natural language querying | None | NLQ translates English to queries | None | **P3** |
| Metric cost/volume management | None | Cost attribution dashboards | Volume dashboards | **P3** |

---

## Phase 1: Foundation (P0) — Close Critical Gaps

These are table-stakes features without which the metrics product is fundamentally limited.

### 1.1 Percentile Aggregations (p50, p75, p90, p95, p99)

**Current**: Only Avg, Sum, Min, Max, Count aggregations.
**Target**: Support percentile aggregations on all metric data, especially histograms and distributions.

**Implementation**:

- Add `P50`, `P75`, `P90`, `P95`, `P99` to the `AggregationType` enum
- For raw metric values: use ClickHouse `quantile(0.50)(value)`, `quantile(0.95)(value)`, etc.
- For histogram data (with `bucketCounts` and `explicitBounds`): implement approximate percentile calculation from bucket data using linear interpolation between bucket boundaries
- Update the metric query builder to include percentile options in the aggregation dropdown
- Update chart rendering to display percentile series

**Files to modify**:
- `Common/Types/BaseDatabase/AggregationType.ts` (add P50, P75, P90, P95, P99)
- `Common/Server/Services/MetricService.ts` (generate quantile SQL)
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricQueryConfig.tsx` (add to dropdown)

### 1.2 Rate/Derivative Calculations

**Current**: No rate or delta computation. Raw cumulative counters are meaningless without rate calculation.
**Target**: Compute per-second rates and deltas from counter/sum metrics.

**Implementation**:

- Add `Rate` and `Delta` as aggregation options
- For cumulative sums: compute `(value_t - value_t-1) / (time_t - time_t-1)` using ClickHouse `runningDifference()`
- Handle counter resets (when value decreases, treat as reset and skip that interval)
- For delta temporality sums: rate is simply `value / interval_seconds`
- Display rate with appropriate units (e.g., "req/s", "bytes/s")

**Files to modify**:
- `Common/Types/BaseDatabase/AggregationType.ts` (add Rate, Delta)
- `Common/Server/Services/MetricService.ts` (generate rate SQL with runningDifference)
- `Common/Types/Metrics/MetricsQuery.ts` (support rate in query config)

### 1.3 Multi-Attribute GROUP BY

**Current**: Single `groupByAttribute: string` field.
**Target**: Group by multiple attributes simultaneously (e.g., by region AND status_code).

**Implementation**:

- Change `groupByAttribute` from `string` to `string[]` in `MetricsQuery`
- Update ClickHouse query generation to GROUP BY multiple extracted JSON attributes
- Update chart rendering to handle multi-dimensional grouping (composite legend labels)
- Update the UI to allow selecting multiple group-by attributes

**Files to modify**:
- `Common/Types/Metrics/MetricsQuery.ts` (change type)
- `Common/Server/Services/MetricService.ts` (update query generation)
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricQueryConfig.tsx` (multi-select UI)
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricGraph.tsx` (composite legends)

### 1.4 Rollups / Downsampling

**Current**: Raw data only with 15-day default TTL. No rollups means long-range queries are slow and historical analysis is limited.
**Target**: Pre-aggregated rollups at multiple resolutions with extended retention.

**Implementation**:

- Create ClickHouse materialized views for automatic rollup:
  ```
  Raw Data (1s resolution) -> 15-day retention
    |-> Materialized View -> 1-min rollups -> 90-day retention
    |-> Materialized View -> 1-hour rollups -> 13-month retention
    |-> Materialized View -> 1-day rollups -> 3-year retention
  ```
- Each rollup table stores: min, max, sum, count, avg, and quantile sketches per metric name + attributes
- Route queries based on time range:
  - < 6 hours: raw data
  - 6 hours - 7 days: 1-min rollups
  - 7 days - 30 days: 1-hour rollups
  - 30+ days: 1-day rollups
- Automatic query routing in the metric service layer

**Files to modify**:
- `Common/Models/AnalyticsModels/MetricRollup1Min.ts` (new)
- `Common/Models/AnalyticsModels/MetricRollup1Hour.ts` (new)
- `Common/Models/AnalyticsModels/MetricRollup1Day.ts` (new)
- `Common/Server/Services/MetricService.ts` (query routing by time range)
- `Worker/DataMigrations/` (new migration to create materialized views)

---

## Phase 2: Visualization & UX (P1) — Match Industry Standard

### 2.1 More Chart Types

**Current**: Line and bar charts only.
**Target**: Add Heatmap, Stacked Area, Pie/Donut, Scatter, Single-Value Billboard, and Gauge.

**Implementation**:

- **Heatmap**: Essential for histogram/distribution data. Use a heatmap library that renders time on X-axis, bucket values on Y-axis, and color intensity for count
- **Stacked Area**: Extension of existing line chart with fill and stacking
- **Pie/Donut**: For showing proportional breakdowns (e.g., request distribution by service)
- **Scatter**: For correlation analysis between two metrics
- **Billboard**: Large single-value display with configurable thresholds for color coding (green/yellow/red)
- **Gauge**: Circular gauge showing a value against a min/max range

**Files to modify**:
- `Common/Types/Dashboard/Chart/ChartType.ts` (add new chart types)
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricGraph.tsx` (render new chart types)
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricCharts.tsx` (chart type selector)

### 2.2 Time-Over-Time Comparison

**Current**: No comparison capability.
**Target**: Overlay current metric data with data from a previous period (1h ago, 1d ago, 1w ago).

**Implementation**:

- Add a "Compare with" dropdown in the metric explorer toolbar (options: 1 hour ago, 1 day ago, 1 week ago, custom)
- Execute the same query twice with shifted time ranges
- Render the comparison series as a dashed/translucent overlay on the same chart
- Show the delta (absolute and percentage) in tooltips

**Files to modify**:
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricExplorer.tsx` (add compare dropdown)
- `Common/Types/Metrics/MetricsQuery.ts` (add compareWith field)
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricGraph.tsx` (render comparison series)

### 2.4 Dashboard Templates

**Current**: No templates.
**Target**: Pre-built dashboards for common scenarios that auto-populate based on detected metrics.

**Implementation**:

- Create MetricsViewConfig templates for:
  - HTTP Service Health (request rate, error rate, latency percentiles)
  - Database Performance (query duration, connection pool, error rate)
  - Kubernetes Metrics (CPU, memory, pod restarts, network)
  - Host Metrics (CPU, memory, disk, network)
  - Runtime Metrics (GC, heap, threads - per language)
- Auto-detect which templates are relevant based on ingested metric names
- "One-click apply" creates a dashboard from the template

**Files to modify**:
- `Common/Types/Metrics/DashboardTemplates/` (new directory with template definitions)
- `App/FeatureSet/Dashboard/src/Pages/Dashboards/Templates.tsx` (new - template gallery)

### 2.5 Summary Metric Type Support

**Current**: Summary type not supported.
**Target**: Ingest and store Summary metrics from OTLP.

**Implementation**:

- Add `Summary` to the metric point type enum
- Store quantile values from summary data points
- Display summary quantiles in the metric explorer

**Files to modify**:
- `Telemetry/Services/OtelMetricsIngestService.ts` (handle summary type)
- `Common/Models/AnalyticsModels/Metric.ts` (add summary-specific columns if needed)

---

## Phase 3: Alerting & Intelligence (P1-P2) — Smart Monitoring

### 3.1 Anomaly Detection

**Current**: Static threshold alerting only.
**Target**: Detect metrics deviating from expected patterns using statistical methods.

**Implementation**:

- Start with rolling mean + N standard deviations (configurable sensitivity: low/medium/high)
- Account for daily/weekly seasonality by comparing to same-time-last-week baselines
- Store baselines in ClickHouse (periodic computation job, hourly)
- Baseline table: metric name, service, hour_of_week, mean, stddev
- On each evaluation: compare current value to baseline, alert if deviation > configured sigma
- Surface anomalies as visual highlights on metric charts (shaded band showing expected range)

**Files to modify**:
- `Common/Models/AnalyticsModels/MetricBaseline.ts` (new - baseline storage)
- `Worker/Jobs/Metrics/ComputeMetricBaselines.ts` (new - periodic baseline computation)
- `Common/Server/Utils/Monitor/Criteria/MetricMonitorCriteria.ts` (add anomaly detection)
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricGraph.tsx` (render anomaly bands)

### 3.2 SLO/SLI Tracking

**Current**: No SLO support.
**Target**: Define Service Level Objectives based on metric queries, track attainment over rolling windows, calculate error budgets.

**Implementation**:

- Create `SLO` PostgreSQL model:
  - Name, description, target percentage (e.g., 99.9%)
  - SLI definition: good events query / total events query (both metric queries)
  - Time window: 7-day, 28-day, or 30-day rolling
  - Alert thresholds: error budget remaining %, burn rate
- SLO dashboard page showing:
  - Current attainment vs target (e.g., 99.85% / 99.9%)
  - Error budget remaining (absolute and percentage)
  - Burn rate chart (current burn rate vs sustainable burn rate)
  - SLI time series chart
- Alert when error budget drops below threshold or burn rate exceeds sustainable rate
- Integrate with existing monitor/incident system

**Files to modify**:
- `Common/Models/DatabaseModels/SLO.ts` (new)
- `Common/Server/Services/SLOService.ts` (new - SLI computation, budget calculation)
- `Worker/Jobs/SLO/EvaluateSLOs.ts` (new - periodic SLO evaluation)
- `App/FeatureSet/Dashboard/src/Pages/SLO/` (new - SLO list, detail, creation pages)

### 3.3 Metric Correlations

**Current**: No correlation capability.
**Target**: When an anomaly is detected, automatically identify other metrics that changed around the same time.

**Implementation**:

- When an anomaly is detected on a metric, query all metrics for the same service/project in the surrounding time window (e.g., +/- 30 minutes)
- Compute Pearson correlation coefficient between the anomalous metric and each candidate
- Rank by absolute correlation value
- Surface top 5-10 correlated metrics in the alert/incident view
- Show correlation chart: anomalous metric overlaid with top correlated metrics

**Files to modify**:
- `Common/Server/Services/MetricCorrelationService.ts` (new)
- `App/FeatureSet/Dashboard/src/Components/Metrics/CorrelatedMetrics.tsx` (new - correlation view)

---

## Phase 4: Scale & Power Features (P2-P3) — Differentiation

### 4.1 Cardinality Management

**Current**: No cardinality visibility or controls.
**Target**: Track unique series count, alert on spikes, allow attribute allowlist/blocklist.

**Implementation**:

- Track unique series count per metric name (via periodic ClickHouse `uniq()` queries)
- Store in a dedicated cardinality tracking table
- Dashboard showing: top metrics by cardinality, cardinality trend over time, per-attribute breakdown
- Allow configuring attribute allowlists/blocklists per metric (applied at ingest time)
- Alert when cardinality exceeds configured budget

**Files to modify**:
- `Worker/Jobs/Metrics/TrackMetricCardinality.ts` (new - periodic cardinality computation)
- `Common/Models/DatabaseModels/MetricCardinalityConfig.ts` (new - allowlist/blocklist)
- `Telemetry/Services/OtelMetricsIngestService.ts` (apply attribute filtering)
- `App/FeatureSet/Dashboard/src/Pages/Settings/MetricCardinality.tsx` (new - cardinality dashboard)

### 4.2 Query Language

**Current**: Form-based UI only.
**Target**: Text-based metrics query language inspired by PromQL/NRQL for advanced users.

**Implementation**:

- Define a grammar supporting:
  ```
  metric_name{attribute="value", attribute2=~"regex"}
    | aggregation(duration)
    by (attribute1, attribute2)
  ```
- Build a parser that translates to the existing ClickHouse query builder
- Offer both UI builder and text modes (toggle like New Relic's basic/advanced)
- Syntax highlighting and autocomplete in the text editor (metric names, attribute keys, functions)
- Functions: `rate()`, `delta()`, `avg()`, `sum()`, `min()`, `max()`, `p50()`, `p95()`, `p99()`, `count()`, `topk()`, `bottomk()`

**Files to modify**:
- `Common/Utils/Metrics/MetricsQueryLanguage.ts` (new - parser and translator)
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricQueryEditor.tsx` (new - text editor with autocomplete)

### 4.3 Golden Signals Dashboards

**Current**: No auto-generated dashboards.
**Target**: Auto-generated dashboards showing Latency, Traffic, Errors, Saturation for each service.

**Implementation**:

- Detect common OpenTelemetry metric names per service:
  - Latency: `http.server.duration`, `http.server.request.duration`
  - Traffic: `http.server.request.count`, `http.server.active_requests`
  - Errors: `http.server.request.count` where status_code >= 500
  - Saturation: `process.runtime.*.memory`, `system.cpu.utilization`
- Auto-create a Golden Signals dashboard for each service with detected metrics
- Include default alert thresholds

**Files to modify**:
- `Worker/Jobs/Metrics/GenerateGoldenSignalsDashboards.ts` (new)
- `Common/Utils/Metrics/GoldenSignalsDetector.ts` (new - metric name pattern matching)

### 4.4 Predictive Alerting

**Current**: No forecasting capability.
**Target**: Forecast metric values and alert before thresholds are breached.

**Implementation**:

- Use linear regression or Holt-Winters on recent data to project forward
- Alert if projected value crosses threshold within configurable forecast window (e.g., "disk full in 4 hours")
- Particularly valuable for capacity planning metrics (disk, memory, connection pools)
- Show forecast as a dashed line extension on metric charts

**Files to modify**:
- `Common/Server/Utils/Monitor/Criteria/MetricMonitorCriteria.ts` (add predictive evaluation)
- `Common/Utils/Metrics/Forecasting.ts` (new - regression/Holt-Winters)
- `App/FeatureSet/Dashboard/src/Components/Metrics/MetricGraph.tsx` (render forecast line)

---

## ClickHouse Storage Improvements

### S.4 Evaluate Map Type for Attributes (MEDIUM)

**Current**: Attributes stored as JSON.
**Target**: Evaluate `Map(LowCardinality(String), String)` for faster attribute-based filtering.

### S.5 Upgrade count/bucketCounts to Int64 (LOW)

**Current**: `Int32` for count and `Array(Int32)` for bucketCounts.
**Target**: `Int64` / `Array(Int64)` to prevent overflow in high-throughput systems.

---

## Quick Wins (Can Ship This Week)

1. **Add p50/p95/p99 to aggregation dropdown** - ClickHouse `quantile()` is straightforward to add
2. **Extend default retention** - 15 days is too short; increase default to 30 days
3. **Multi-attribute GROUP BY** - Change `groupByAttribute: string` to `groupByAttribute: string[]`
4. **Add stacked area chart type** - Simple extension of existing line chart

---

## Recommended Implementation Order

1. **Quick Wins** - Ship p50/p95/p99, multi-attribute GROUP BY, stacked area
2. **Phase 1.1** - Percentile aggregations (full implementation beyond quick win)
3. **Phase 1.2** - Rate/derivative calculations
4. **Phase 1.4** - Rollups/downsampling (enables long-range queries)
5. **Phase 2.1** - More chart types (heatmap, pie, gauge, billboard)
6. **Phase 2.2** - Time-over-time comparison
7. **Phase 1.3** - Multi-attribute GROUP BY (full implementation)
8. **Phase 3.1** - Anomaly detection
9. **Phase 3.2** - SLO/SLI tracking
10. **Phase 2.4** - Dashboard templates
11. **Phase 4.1** - Cardinality management
12. **Phase 4.2** - Query language
13. **Phase 4.3** - Golden Signals dashboards
14. **Phase 4.4** - Predictive alerting
15. **Phase 3.3** - Metric correlations

## Verification

For each feature:
1. Unit tests for new aggregation types, rate calculations, unit formatting, query language parser
2. Integration tests for new API endpoints (percentiles, rollup queries, SLO evaluation)
3. Manual verification via the dev server at `https://oneuptimedev.genosyn.com/dashboard/{projectId}/metrics`
4. Check ClickHouse query performance with `EXPLAIN` for new query patterns
5. Verify rollup accuracy by comparing rollup results to raw data results for overlapping time ranges
6. Load test cardinality tracking and anomaly detection jobs to ensure they don't impact ingestion
