github/OpenArchiver
1
1
Fork 0
mirror of https://github.com/LogicLabs-OU/OpenArchiver.git synced 2026-04-06 00:31:57 +02:00
Code Issues 161 Packages Projects Releases Wiki Activity

retention policy (backend/frontend)

Browse Source
This commit is contained in:
wayneshn
2026-03-09 18:25:09 +01:00
parent d1615b9510
commit 3a30d22caa
19 changed files with 3443 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

267
docs/enterprise/retention-policy/api.md Normal file
View File

@@ -0,0 +1,267 @@
# Retention Policy: API Endpoints
The retention policy feature exposes a RESTful API for managing retention policies and simulating policy evaluation against email metadata. All endpoints require authentication and the `manage:all` permission.
**Base URL:** `/api/v1/enterprise/retention-policy`
All endpoints also require the `RETENTION_POLICY` feature to be enabled in the enterprise license.
---
## List All Policies
Retrieves all retention policies, ordered by priority ascending.
- **Endpoint:** `GET /policies`
- **Method:** `GET`
- **Authentication:** Required
- **Permission:** `manage:all`
### Response Body
```json
[
{
"id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"name": "Default 7-Year Retention",
"description": "Retain all emails for 7 years per regulatory requirements.",
"priority": 1,
"conditions": null,
"ingestionScope": null,
"retentionPeriodDays": 2555,
"isActive": true,
"createdAt": "2025-10-01T00:00:00.000Z",
"updatedAt": "2025-10-01T00:00:00.000Z"
}
]
```
---
## Get Policy by ID
Retrieves a single retention policy by its UUID.
- **Endpoint:** `GET /policies/:id`
- **Method:** `GET`
- **Authentication:** Required
- **Permission:** `manage:all`
### Path Parameters
| Parameter | Type | Description |
| --------- | ------ | ------------------------------ |
| `id` | `uuid` | The UUID of the policy to get. |
### Response Body
Returns a single policy object (same shape as the list endpoint), or `404` if not found.
---
## Create Policy
Creates a new retention policy. The policy name must be unique across the system.
- **Endpoint:** `POST /policies`
- **Method:** `POST`
- **Authentication:** Required
- **Permission:** `manage:all`
### Request Body
| Field | Type | Required | Description |
| ------------------- | --------------------- | -------- | ---------------------------------------------------------------------------------------------- |
| `name` | `string` | Yes | Unique policy name. Max 255 characters. |
| `description` | `string` | No | Human-readable description. Max 1000 characters. |
| `priority` | `integer` | Yes | Positive integer. Lower values indicate higher priority. |
| `retentionPeriodDays` | `integer` | Yes | Number of days to retain matching emails. Minimum 1. |
| `actionOnExpiry` | `string` | Yes | Action to take when the retention period expires. Currently only `"delete_permanently"`. |
| `isEnabled` | `boolean` | No | Whether the policy is active. Defaults to `true`. |
| `conditions` | `RuleGroup \| null` | No | Condition rules for targeting specific emails. `null` matches all emails. |
| `ingestionScope` | `string[] \| null` | No | Array of ingestion source UUIDs to scope the policy to. `null` applies to all sources. |
#### Conditions (RuleGroup) Schema
```json
{
"logicalOperator": "AND",
"rules": [
{
"field": "sender",
"operator": "domain_match",
"value": "example.com"
},
{
"field": "subject",
"operator": "contains",
"value": "invoice"
}
]
}
```
**Supported fields:** `sender`, `recipient`, `subject`, `attachment_type`
**Supported operators:**
| Operator | Description |
| -------------- | ------------------------------------------------------------------ |
| `equals` | Exact case-insensitive match. |
| `not_equals` | Inverse of `equals`. |
| `contains` | Case-insensitive substring match. |
| `not_contains` | Inverse of `contains`. |
| `starts_with` | Case-insensitive prefix match. |
| `ends_with` | Case-insensitive suffix match. |
| `domain_match` | Matches when an email address ends with `@<value>`. |
| `regex_match` | ECMAScript regex (case-insensitive). Max pattern length: 200 chars.|
**Validation limits:**
- Maximum 50 rules per group.
- Rule `value` must be between 1 and 500 characters.
### Example Request
```json
{
"name": "Finance Department - 10 Year",
"description": "Extended retention for finance-related correspondence.",
"priority": 2,
"retentionPeriodDays": 3650,
"actionOnExpiry": "delete_permanently",
"conditions": {
"logicalOperator": "OR",
"rules": [
{
"field": "sender",
"operator": "domain_match",
"value": "finance.acme.com"
},
{
"field": "recipient",
"operator": "domain_match",
"value": "finance.acme.com"
}
]
},
"ingestionScope": ["b2c3d4e5-f6a7-8901-bcde-f23456789012"]
}
```
### Response
- **`201 Created`** — Returns the created policy object.
- **`409 Conflict`** — A policy with this name already exists.
- **`422 Unprocessable Entity`** — Validation errors.
---
## Update Policy
Updates an existing retention policy. Only the fields included in the request body are modified.
- **Endpoint:** `PUT /policies/:id`
- **Method:** `PUT`
- **Authentication:** Required
- **Permission:** `manage:all`
### Path Parameters
| Parameter | Type | Description |
| --------- | ------ | --------------------------------- |
| `id` | `uuid` | The UUID of the policy to update. |
### Request Body
All fields from the create endpoint are accepted, and all are optional. Only provided fields are updated.
To clear conditions (make the policy match all emails), send `"conditions": null`.
To clear ingestion scope (make the policy apply to all sources), send `"ingestionScope": null`.
### Response
- **`200 OK`** — Returns the updated policy object.
- **`404 Not Found`** — Policy with the given ID does not exist.
- **`422 Unprocessable Entity`** — Validation errors.
---
## Delete Policy
Permanently deletes a retention policy. This action is irreversible.
- **Endpoint:** `DELETE /policies/:id`
- **Method:** `DELETE`
- **Authentication:** Required
- **Permission:** `manage:all`
### Path Parameters
| Parameter | Type | Description |
| --------- | ------ | --------------------------------- |
| `id` | `uuid` | The UUID of the policy to delete. |
### Response
- **`204 No Content`** — Policy successfully deleted.
- **`404 Not Found`** — Policy with the given ID does not exist.
---
## Evaluate Email (Policy Simulator)
Evaluates a set of email metadata against all active policies and returns the applicable retention period and matching policy IDs. This endpoint does not modify any data — it is a read-only simulation tool.
- **Endpoint:** `POST /policies/evaluate`
- **Method:** `POST`
- **Authentication:** Required
- **Permission:** `manage:all`
### Request Body
| Field | Type | Required | Description |
| ---------------------------------- | ---------- | -------- | -------------------------------------------------------- |
| `emailMetadata.sender` | `string` | Yes | Sender email address. Max 500 characters. |
| `emailMetadata.recipients` | `string[]` | Yes | Recipient email addresses. Max 500 entries. |
| `emailMetadata.subject` | `string` | Yes | Email subject line. Max 2000 characters. |
| `emailMetadata.attachmentTypes` | `string[]` | Yes | File extensions (e.g., `[".pdf", ".xml"]`). Max 100. |
| `emailMetadata.ingestionSourceId` | `uuid` | No | Optional ingestion source UUID for scope-aware evaluation.|
### Example Request
```json
{
"emailMetadata": {
"sender": "cfo@finance.acme.com",
"recipients": ["legal@acme.com"],
"subject": "Q4 Invoice Reconciliation",
"attachmentTypes": [".pdf", ".xlsx"],
"ingestionSourceId": "b2c3d4e5-f6a7-8901-bcde-f23456789012"
}
}
```
### Response Body
```json
{
"appliedRetentionDays": 3650,
"actionOnExpiry": "delete_permanently",
"matchingPolicyIds": [
"a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"c3d4e5f6-a7b8-9012-cdef-345678901234"
]
}
```
| Field | Type | Description |
| ---------------------- | ---------- | ------------------------------------------------------------------------------------- |
| `appliedRetentionDays` | `integer` | The longest retention period from all matching policies. `0` means no policy matched. |
| `actionOnExpiry` | `string` | The action to take on expiry. Currently always `"delete_permanently"`. |
| `matchingPolicyIds` | `string[]` | UUIDs of all policies that matched the provided metadata. |
### Response Codes
- **`200 OK`** — Evaluation completed.
- **`422 Unprocessable Entity`** — Validation errors in the request body.

93
docs/enterprise/retention-policy/guide.md Normal file
View File

@@ -0,0 +1,93 @@
# Retention Policy: User Interface
The retention policy management interface is located at **Dashboard → Compliance → Retention Policies**. It provides a comprehensive view of all configured policies and tools for creating, editing, deleting, and simulating retention rules.
## Policy Table
The main page displays a table of all retention policies with the following columns:
- **Name:** The policy name and its UUID displayed underneath for reference.
- **Priority:** The numeric priority value. Lower values indicate higher priority.
- **Retention Period:** The number of days emails matching this policy are retained before expiry.
- **Ingestion Scope:** Shows which ingestion sources the policy is restricted to. Displays "All ingestion sources" when the policy has no scope restriction, or individual source name badges when scoped.
- **Conditions:** A summary of the rule group. Displays "No conditions (matches all emails)" for policies without conditions, or "N rule(s) (AND/OR)" for policies with conditions.
- **Status:** A badge indicating whether the policy is Active or Inactive.
- **Actions:** Edit and Delete buttons for each policy.
The table is sorted by policy priority by default.
## Creating a Policy
Click the **"Create Policy"** button above the table to open the creation dialog. The form contains the following sections:
### Basic Information
- **Policy Name:** A unique, descriptive name for the policy.
- **Description:** An optional detailed description of the policy's purpose.
- **Priority:** A positive integer determining evaluation order (lower = higher priority).
- **Retention Period (Days):** The number of days to retain matching emails.
### Ingestion Scope
This section controls which ingestion sources the policy applies to:
- **"All ingestion sources" toggle:** When enabled, the policy applies to emails from all ingestion sources. This is the default.
- **Per-source checkboxes:** When the "all" toggle is disabled, individual ingestion sources can be selected. Each source displays its name and provider type as a badge.
### Condition Rules
Conditions define which emails the policy targets. If no conditions are added, the policy matches all emails (within its ingestion scope).
- **Logical Operator:** Choose **AND** (all rules must match) or **OR** (any rule must match).
- **Add Rule:** Each rule consists of:
- **Field:** The email metadata field to evaluate (`sender`, `recipient`, `subject`, or `attachment_type`).
- **Operator:** The comparison operator (see [Supported Operators](#supported-operators) below).
- **Value:** The string value to compare against.
- **Remove Rule:** Each rule has a remove button to delete it from the group.
### Supported Operators
| Operator | Display Name | Description |
| -------------- | ------------- | ----------------------------------------------------------- |
| `equals` | Equals | Exact case-insensitive match. |
| `not_equals` | Not Equals | Inverse of equals. |
| `contains` | Contains | Case-insensitive substring match. |
| `not_contains` | Not Contains | Inverse of contains. |
| `starts_with` | Starts With | Case-insensitive prefix match. |
| `ends_with` | Ends With | Case-insensitive suffix match. |
| `domain_match` | Domain Match | Matches when an email address ends with `@<value>`. |
| `regex_match` | Regex Match | ECMAScript regular expression (case-insensitive, max 200 chars). |
### Policy Status
- **Enable Policy toggle:** Controls whether the policy is active immediately upon creation.
## Editing a Policy
Click the **Edit** button (pencil icon) on any policy row to open the edit dialog. The form is pre-populated with the policy's current values. All fields can be modified, and the same validation rules apply as during creation.
## Deleting a Policy
Click the **Delete** button (trash icon) on any policy row. A confirmation dialog appears to prevent accidental deletion. Deleting a policy is irreversible. Once deleted, the policy no longer affects the lifecycle worker's evaluation of emails.
## Policy Simulator
The **"Simulate Policy"** button opens a simulation tool that evaluates hypothetical email metadata against all active policies without making any changes.
### Simulator Input Fields
- **Sender Email:** The sender address to evaluate (e.g., `cfo@finance.acme.com`).
- **Recipients:** A comma-separated list of recipient email addresses.
- **Subject:** The email subject line.
- **Attachment Types:** A comma-separated list of file extensions (e.g., `.pdf, .xlsx`).
- **Ingestion Source:** An optional dropdown to select a specific ingestion source for scope-aware evaluation. Defaults to "All sources".
### Simulator Results
After submission, the simulator displays:
- **Applied Retention Period:** The longest retention period from all matching policies, displayed in days.
- **Action on Expiry:** The action that would be taken when the retention period expires (currently always "Permanent Deletion").
- **Matching Policies:** A list of all policy IDs (with their names) that matched the provided metadata. If no policies match, a message indicates that no matching policies were found.
The simulator is a safe, read-only tool intended for testing and verifying policy configurations before they affect live data.

55
docs/enterprise/retention-policy/index.md Normal file
View File

@@ -0,0 +1,55 @@
# Retention Policy
The Retention Policy Engine is an enterprise-grade feature that automates the lifecycle management of archived emails. It enables organizations to define time-based retention rules that determine how long archived emails are kept before they are permanently deleted, ensuring compliance with data protection regulations and internal data governance policies.
## Core Principles
### 1. Policy-Based Automation
Email deletion is never arbitrary. Every deletion is governed by one or more explicitly configured retention policies that define the retention period in days, the conditions under which the policy applies, and the action to take when an email expires. The lifecycle worker processes emails in batches on a recurring schedule, ensuring continuous enforcement without manual intervention.
### 2. Condition-Based Targeting
Policies can target specific subsets of archived emails using a flexible condition builder. Conditions are evaluated against email metadata fields (sender, recipient, subject, attachment type) using a variety of string-matching operators. Conditions within a policy are grouped using AND/OR logic, allowing precise control over which emails a policy applies to.
### 3. Ingestion Scope
Each policy can optionally be scoped to one or more ingestion sources. When an ingestion scope is set, the policy only applies to emails that were archived from those specific sources. Policies with no ingestion scope (null) apply to all emails regardless of their source.
### 4. Priority and Max-Duration-Wins
When multiple policies match a single email, the system applies **max-duration-wins** logic: the longest matching retention period is used. This ensures that if any policy requires an email to be kept longer, that requirement is honored. The priority field on each policy provides an ordering mechanism for administrative purposes and future conflict-resolution enhancements.
### 5. Full Audit Trail
Every policy lifecycle event — creation, modification, deletion, and every automated email deletion — is recorded in the immutable [Audit Log](../audit-log/index.md). Automated deletions include the IDs of the governing policies in the audit log entry, ensuring full traceability from deletion back to the rule that triggered it.
### 6. Fail-Safe Behavior
The system is designed to err on the side of caution:
- If no policy matches an email, the email is **not** deleted.
- If the lifecycle worker encounters an error processing a specific email, it logs the error and continues with the remaining emails in the batch.
- Invalid regex patterns in `regex_match` rules are treated as non-matching rather than causing failures.
## Feature Requirements
The Retention Policy Engine requires:
- An active **Enterprise license** with the `RETENTION_POLICY` feature enabled.
- The `manage:all` permission for the authenticated user to access the policy management API and UI.
## Architecture Overview
The feature is composed of the following components:
| Component | Location | Description |
| -------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------ |
| Types | `packages/types/src/retention.types.ts` | Shared TypeScript types for policies, rules, and evaluation. |
| Database Schema | `packages/backend/src/database/schema/compliance.ts` | Drizzle ORM table definition for `retention_policies`. |
| Retention Service | `packages/enterprise/src/modules/retention-policy/RetentionService.ts`| CRUD operations and the evaluation engine. |
| API Controller | `packages/enterprise/src/modules/retention-policy/retention-policy.controller.ts` | Express request handlers with Zod validation. |
| API Routes | `packages/enterprise/src/modules/retention-policy/retention-policy.routes.ts` | Route registration with auth and feature guards. |
| Module | `packages/enterprise/src/modules/retention-policy/retention-policy.module.ts` | Enterprise module bootstrap. |
| Lifecycle Worker | `packages/enterprise/src/workers/lifecycle.worker.ts` | BullMQ worker for automated retention enforcement. |
| Frontend Page | `packages/frontend/src/routes/dashboard/compliance/retention-policies/` | SvelteKit page for policy management and simulation. |

106
docs/enterprise/retention-policy/lifecycle-worker.md Normal file
View File

@@ -0,0 +1,106 @@
# Retention Policy: Lifecycle Worker
The lifecycle worker is the automated enforcement component of the retention policy engine. It runs as a BullMQ background worker that periodically scans all archived emails, evaluates them against active retention policies, and permanently deletes emails that have exceeded their retention period.
## Location
`packages/enterprise/src/workers/lifecycle.worker.ts`
## How It Works
### Scheduling
The lifecycle worker is registered as a repeatable BullMQ cron job on the `compliance-lifecycle` queue. It is scheduled to run daily at **02:00 UTC** by default. The cron schedule is configured via:
```typescript
repeat: { pattern: '0 2 * * *' } // daily at 02:00 UTC
```
The `scheduleLifecycleJob()` function is called once during enterprise application startup to register the repeatable job with BullMQ.
### Batch Processing
To avoid loading the entire `archived_emails` table into memory, the worker processes emails in configurable batches:
1. **Batch size** is controlled by the `RETENTION_BATCH_SIZE` environment variable.
2. Emails are ordered by `archivedAt` ascending.
3. The worker iterates through batches using offset-based pagination until an empty batch is returned, indicating all emails have been processed.
### Per-Email Processing Flow
For each email in a batch, the worker:
1. **Extracts metadata:** Builds a `PolicyEvaluationRequest` from the email's database record:
- `sender`: The sender email address.
- `recipients`: All To, CC, and BCC recipient addresses.
- `subject`: The email subject line.
- `attachmentTypes`: File extensions (e.g., `.pdf`) extracted from attachment filenames via a join query.
- `ingestionSourceId`: The UUID of the ingestion source that archived this email.
2. **Evaluates policies:** Passes the metadata to `RetentionService.evaluateEmail()`, which returns:
- `appliedRetentionDays`: The longest matching retention period (0 if no policy matches).
- `matchingPolicyIds`: UUIDs of all matching policies.
3. **Checks for expiry:**
- If `appliedRetentionDays === 0`, no policy matched — the email is **skipped** (not deleted).
- Otherwise, the email's age is calculated from its `sentAt` date.
- If the age in days exceeds `appliedRetentionDays`, the email has expired.
4. **Deletes expired emails:** Calls `ArchivedEmailService.deleteArchivedEmail()` with:
- `systemDelete: true` — Bypasses the `ENABLE_DELETION` configuration guard so retention enforcement always works regardless of that global setting.
- `governingRule` — A string listing the matching policy IDs for the audit log entry (e.g., `"Policy IDs: abc-123, def-456"`).
5. **Logs the deletion:** A structured log entry records the email ID and its age in days.
### Error Handling
If processing a specific email fails (e.g., due to a database error or storage issue), the error is logged and the worker continues to the next email in the batch. This ensures that a single problematic email does not block the processing of the remaining emails.
If the entire job fails, BullMQ records the failure and the job ID and error are logged. Failed jobs are retained (up to 50) for debugging.
## System Actor
Automated deletions are attributed to a synthetic system actor in the audit log:
| Field | Value |
| ------------ | ------------------------------------ |
| ID | `system:lifecycle-worker` |
| Email | `system@open-archiver.internal` |
| Name | System Lifecycle Worker |
| Actor IP | `system` |
This well-known identifier can be filtered in the [Audit Log](../audit-log/index.md) to view all retention-based deletions.
## Audit Trail
Every email deleted by the lifecycle worker produces an audit log entry with:
- **Action type:** `DELETE`
- **Target type:** `ArchivedEmail`
- **Target ID:** The UUID of the deleted email
- **Actor:** `system:lifecycle-worker`
- **Details:** Includes `reason: "RetentionExpiration"` and `governingRule` listing the matching policy IDs
This ensures that every automated deletion is fully traceable back to the specific policies that triggered it.
## Configuration
| Environment Variable | Description | Default |
| ------------------------- | ---------------------------------------------------- | ------- |
| `RETENTION_BATCH_SIZE` | Number of emails to process per batch iteration. | — |
## BullMQ Worker Settings
| Setting | Value | Description |
| -------------------- | ---------------------- | -------------------------------------------------- |
| Queue name | `compliance-lifecycle` | The BullMQ queue name. |
| Job ID | `lifecycle-daily` | Stable job ID for the repeatable cron job. |
| `removeOnComplete` | Keep last 10 | Completed jobs retained for monitoring. |
| `removeOnFail` | Keep last 50 | Failed jobs retained for debugging. |
## Integration with Deletion Guard
The core `ArchivedEmailService.deleteArchivedEmail()` method includes a deletion guard controlled by the `ENABLE_DELETION` system setting. When called with `systemDelete: true`, the lifecycle worker bypasses this guard. This design ensures that:
- Manual user deletions can be disabled organization-wide via the system setting.
- Automated retention enforcement always operates regardless of that setting, because retention compliance is a legal obligation that cannot be paused by a UI toggle.

138
docs/enterprise/retention-policy/retention-service.md Normal file
View File

@@ -0,0 +1,138 @@
# Retention Policy: Backend Implementation
The backend implementation of the retention policy engine is handled by the `RetentionService`, located in `packages/enterprise/src/modules/retention-policy/RetentionService.ts`. This service encapsulates all CRUD operations for policies and the core evaluation engine that determines which policies apply to a given email.
## Database Schema
The `retention_policies` table is defined in `packages/backend/src/database/schema/compliance.ts` using Drizzle ORM:
| Column | Type | Description |
| --------------------- | -------------------------- | --------------------------------------------------------------------------- |
| `id` | `uuid` (PK) | Auto-generated unique identifier. |
| `name` | `text` (unique, not null) | Human-readable policy name. |
| `description` | `text` | Optional description. |
| `priority` | `integer` (not null) | Priority for ordering. Lower = higher priority. |
| `retention_period_days` | `integer` (not null) | Number of days to retain matching emails. |
| `action_on_expiry` | `enum` (not null) | Action on expiry (`delete_permanently`). |
| `is_enabled` | `boolean` (default: true) | Whether the policy is active. |
| `conditions` | `jsonb` | Serialized `RetentionRuleGroup` or null (null = matches all). |
| `ingestion_scope` | `jsonb` | Array of ingestion source UUIDs or null (null = all sources). |
| `created_at` | `timestamptz` | Creation timestamp. |
| `updated_at` | `timestamptz` | Last update timestamp. |
## CRUD Operations
The `RetentionService` class provides the following methods:
### `createPolicy(data, actorId, actorIp)`
Inserts a new policy into the database and creates an audit log entry with action type `CREATE` and target type `RetentionPolicy`. The audit log details include the policy name, retention period, priority, action on expiry, and ingestion scope.
### `getPolicies()`
Returns all policies ordered by priority ascending. The raw database rows are mapped through `mapDbPolicyToType()`, which converts the DB column `isEnabled` to the shared type field `isActive` and normalizes date fields to ISO strings.
### `getPolicyById(id)`
Returns a single policy by UUID, or null if not found.
### `updatePolicy(id, data, actorId, actorIp)`
Partially updates a policy — only fields present in the DTO are modified. The `updatedAt` timestamp is always set to the current time. An audit log entry is created with action type `UPDATE`, recording which fields were changed.
Throws an error if the policy is not found.
### `deletePolicy(id, actorId, actorIp)`
Deletes a policy by UUID and creates an audit log entry with action type `DELETE`, recording the deleted policy's name. Returns `false` if the policy was not found.
## Evaluation Engine
The evaluation engine is the core logic that determines which policies apply to a given email. It is used by both the lifecycle worker (for automated enforcement) and the policy simulator endpoint (for testing).
### `evaluateEmail(metadata)`
This is the primary evaluation method. It accepts email metadata and returns:
- `appliedRetentionDays`: The longest matching retention period (max-duration-wins).
- `matchingPolicyIds`: UUIDs of all policies that matched.
- `actionOnExpiry`: Always `"delete_permanently"` in the current implementation.
The evaluation flow:
1. **Fetch active policies:** Queries all policies where `isEnabled = true`.
2. **Ingestion scope check:** For each policy with a non-null `ingestionScope`, the email's `ingestionSourceId` must be included in the scope array. If not, the policy is skipped.
3. **Condition evaluation:** If the policy has no conditions (`null`), it matches all emails within scope. Otherwise, the condition rule group is evaluated.
4. **Max-duration-wins:** If multiple policies match, the longest `retentionPeriodDays` is used.
5. **Zero means no match:** A return value of `appliedRetentionDays = 0` indicates no policy matched — the lifecycle worker will not delete the email.
### `_evaluateRuleGroup(group, metadata)`
Evaluates a `RetentionRuleGroup` using AND or OR logic:
- **AND:** Every rule in the group must pass.
- **OR:** At least one rule must pass.
- An empty rules array evaluates to `true`.
### `_evaluateRule(rule, metadata)`
Evaluates a single rule against the email metadata. All string comparisons are case-insensitive (both sides are lowercased before comparison). The behavior depends on the field:
| Field | Behavior |
| ----------------- | ------------------------------------------------------------------------ |
| `sender` | Compares against the sender email address. |
| `recipient` | Passes if **any** recipient matches the operator. |
| `subject` | Compares against the email subject. |
| `attachment_type` | Passes if **any** attachment file extension matches (e.g., `.pdf`). |
### `_applyOperator(haystack, operator, needle)`
Applies a string-comparison operator between two pre-lowercased strings:
| Operator | Implementation |
| -------------- | ----------------------------------------------------------------------------- |
| `equals` | `haystack === needle` |
| `not_equals` | `haystack !== needle` |
| `contains` | `haystack.includes(needle)` |
| `not_contains` | `!haystack.includes(needle)` |
| `starts_with` | `haystack.startsWith(needle)` |
| `ends_with` | `haystack.endsWith(needle)` |
| `domain_match` | `haystack.endsWith('@' + needle)` (auto-prepends `@` if missing) |
| `regex_match` | `new RegExp(needle, 'i').test(haystack)` with safety guards (see below) |
### Security: `regex_match` Safeguards
The `regex_match` operator includes protections against Regular Expression Denial of Service (ReDoS):
1. **Length limit:** Patterns exceeding 200 characters (`MAX_REGEX_LENGTH`) are rejected and treated as non-matching. A warning is logged.
2. **Error handling:** Invalid regex syntax is caught in a try/catch block and treated as non-matching. A warning is logged.
3. **Flags:** Only the case-insensitive flag (`i`) is used. Global and multiline flags are excluded to prevent stateful matching bugs.
## Request Validation
The `RetentionPolicyController` (`retention-policy.controller.ts`) validates all incoming requests using Zod schemas before passing data to the service:
| Constraint | Limit |
| --------------------------- | -------------------------------------------------------------- |
| Policy name | 1–255 characters. |
| Description | Max 1000 characters. |
| Priority | Positive integer (≥ 1). |
| Retention period | Positive integer (≥ 1 day). |
| Rules per group | Max 50. |
| Rule value | 1–500 characters. |
| Ingestion scope entries | Each must be a valid UUID. Empty arrays are coerced to `null`. |
| Evaluate — sender | Max 500 characters. |
| Evaluate — recipients | Max 500 entries, each max 500 characters. |
| Evaluate — subject | Max 2000 characters. |
| Evaluate — attachment types | Max 100 entries, each max 50 characters. |
## Module Registration
The `RetentionPolicyModule` (`retention-policy.module.ts`) implements the `ArchiverModule` interface and registers the API routes at:
```
/{api.version}/enterprise/retention-policy
```
All routes are protected by:
1. `requireAuth` — Ensures the request includes a valid authentication token.
2. `featureEnabled(OpenArchiverFeature.RETENTION_POLICY)` — Ensures the enterprise license includes the retention policy feature.
3. `requirePermission('manage', 'all')` — Ensures the user has administrative permissions.

3
packages/backend/src/database/migrations/0025_peaceful_grim_reaper.sql Normal file
View File

@@ -0,0 +1,3 @@
ALTER TYPE "public"."audit_log_target_type" ADD VALUE 'RetentionPolicy' BEFORE 'Role';--> statement-breakpoint
ALTER TYPE "public"."audit_log_target_type" ADD VALUE 'SystemEvent' BEFORE 'SystemSettings';--> statement-breakpoint
ALTER TABLE "retention_policies" ADD COLUMN "ingestion_scope" jsonb DEFAULT 'null'::jsonb;

1560
packages/backend/src/database/migrations/meta/0025_snapshot.json Normal file
View File

File diff suppressed because it is too large Load Diff

7
packages/backend/src/database/migrations/meta/_journal.json
View File

@@ -176,6 +176,13 @@
"when": 1772842674479,
"tag": "0024_careful_black_panther",
"breakpoints": true
},
{
"idx": 25,
"version": "7",
"when": 1773013461190,
"tag": "0025_peaceful_grim_reaper",
"breakpoints": true
}
]
}

6
packages/backend/src/database/schema/compliance.ts
View File

@@ -12,7 +12,6 @@ import {
varchar,
} from 'drizzle-orm/pg-core';
import { archivedEmails } from './archived-emails';
import { custodians } from './custodians';
import { users } from './users';
// --- Enums ---
@@ -33,6 +32,11 @@ export const retentionPolicies = pgTable('retention_policies', {
actionOnExpiry: retentionActionEnum('action_on_expiry').notNull(),
isEnabled: boolean('is_enabled').notNull().default(true),
conditions: jsonb('conditions'),
/**
* Array of ingestion source UUIDs this policy is restricted to.
* null means the policy applies to all ingestion sources.
*/
ingestionScope: jsonb('ingestion_scope').$type<string[] | null>().default(null),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
});

4
packages/backend/src/index.ts
View File

@@ -6,5 +6,7 @@ export * from './services/AuditService';
export * from './api/middleware/requireAuth';
export * from './api/middleware/requirePermission';
export { db } from './database';
export * as drizzleOrm from 'drizzle-orm';
export * from './database/schema';
export { AuditService } from './services/AuditService';
export * from './config'
export * from './jobs/queues'

6
packages/backend/src/jobs/queues.ts
View File

@@ -27,3 +27,9 @@ export const indexingQueue = new Queue('indexing', {
connection,
defaultJobOptions,
});
// Queue for the Data Lifecycle Manager (retention policy enforcement)
export const complianceLifecycleQueue = new Queue('compliance-lifecycle', {
connection,
defaultJobOptions,
});

21
packages/backend/src/services/ArchivedEmailService.ts
View File

@@ -199,7 +199,13 @@ export class ArchivedEmailService {
emailId: string,
actor: User,
actorIp: string,
options: { systemDelete?: boolean } = {}
options: {
systemDelete?: boolean;
/**
* Human-readable name of the retention rule that triggered deletion
*/
governingRule?: string;
} = {}
): Promise<void> {
checkDeletionEnabled({ allowSystemDelete: options.systemDelete });
@@ -270,15 +276,22 @@ export class ArchivedEmailService {
await db.delete(archivedEmails).where(eq(archivedEmails.id, emailId));
// Build audit details: system-initiated deletions carry retention context
// for GoBD compliance; manual deletions record only the reason.
const auditDetails: Record<string, unknown> = {
reason: options.systemDelete ? 'RetentionExpiration' : 'ManualDeletion',
};
if (options.systemDelete && options.governingRule) {
auditDetails.governingRule = options.governingRule;
}
await this.auditService.createAuditLog({
actorIdentifier: actor.id,
actionType: 'DELETE',
targetType: 'ArchivedEmail',
targetId: emailId,
actorIp,
details: {
reason: 'ManualDeletion',
},
details: auditDetails,
});
}
}

366
packages/frontend/src/lib/components/custom/RetentionPolicyForm.svelte Normal file
View File

@@ -0,0 +1,366 @@
<script lang="ts">
import { t } from '$lib/translations';
import { Input } from '$lib/components/ui/input';
import { Label } from '$lib/components/ui/label';
import { Button } from '$lib/components/ui/button';
import { Switch } from '$lib/components/ui/switch';
import { Badge } from '$lib/components/ui/badge';
import * as Select from '$lib/components/ui/select/index.js';
import { enhance } from '$app/forms';
import { Trash2, Plus, Database } from 'lucide-svelte';
import type {
RetentionPolicy,
RetentionRule,
ConditionField,
ConditionOperator,
LogicalOperator,
SafeIngestionSource,
} from '@open-archiver/types';
interface Props {
/** Existing policy to edit; undefined means create mode */
policy?: RetentionPolicy;
isLoading?: boolean;
/** All available ingestion sources for scope selection */
ingestionSources?: SafeIngestionSource[];
/** Form action to target, e.g. '?/create' or '?/update' */
action: string;
onCancel: () => void;
/** Called after successful submission so the parent can close the dialog */
onSuccess: () => void;
}
let {
policy,
isLoading = $bindable(false),
ingestionSources = [],
action,
onCancel,
onSuccess,
}: Props = $props();
// --- Form state ---
let name = $state(policy?.name ?? '');
let description = $state(policy?.description ?? '');
let priority = $state(policy?.priority ?? 10);
let retentionPeriodDays = $state(policy?.retentionPeriodDays ?? 365);
let isEnabled = $state(policy?.isActive ?? true);
// Conditions state
let logicalOperator = $state<LogicalOperator>(
policy?.conditions?.logicalOperator ?? 'AND'
);
let rules = $state<RetentionRule[]>(
policy?.conditions?.rules ? [...policy.conditions.rules] : []
);
// Ingestion scope: set of selected ingestion source IDs
// Empty set = null scope = applies to all
let selectedIngestionIds = $state<Set<string>>(
new Set(policy?.ingestionScope ?? [])
);
// The conditions JSON that gets sent as a hidden form field
const conditionsJson = $derived(JSON.stringify({ logicalOperator, rules }));
// The ingestionScope value: comma-separated UUIDs, or empty string for null (all)
const ingestionScopeValue = $derived(
selectedIngestionIds.size > 0 ? [...selectedIngestionIds].join(',') : ''
);
// --- Field options ---
const fieldOptions: { value: ConditionField; label: string }[] = [
{ value: 'sender', label: $t('app.retention_policies.field_sender') },
{ value: 'recipient', label: $t('app.retention_policies.field_recipient') },
{ value: 'subject', label: $t('app.retention_policies.field_subject') },
{ value: 'attachment_type', label: $t('app.retention_policies.field_attachment_type') },
];
// --- Operator options (grouped for readability) ---
const operatorOptions: { value: ConditionOperator; label: string }[] = [
{ value: 'equals', label: $t('app.retention_policies.operator_equals') },
{ value: 'not_equals', label: $t('app.retention_policies.operator_not_equals') },
{ value: 'contains', label: $t('app.retention_policies.operator_contains') },
{ value: 'not_contains', label: $t('app.retention_policies.operator_not_contains') },
{ value: 'starts_with', label: $t('app.retention_policies.operator_starts_with') },
{ value: 'ends_with', label: $t('app.retention_policies.operator_ends_with') },
{ value: 'domain_match', label: $t('app.retention_policies.operator_domain_match') },
{ value: 'regex_match', label: $t('app.retention_policies.operator_regex_match') },
];
function addRule() {
rules = [...rules, { field: 'sender', operator: 'contains', value: '' }];
}
function removeRule(index: number) {
rules = rules.filter((_, i) => i !== index);
}
function updateRuleField(index: number, field: ConditionField) {
rules = rules.map((r, i) => (i === index ? { ...r, field } : r));
}
function updateRuleOperator(index: number, operator: ConditionOperator) {
rules = rules.map((r, i) => (i === index ? { ...r, operator } : r));
}
function updateRuleValue(index: number, value: string) {
rules = rules.map((r, i) => (i === index ? { ...r, value } : r));
}
function toggleIngestionSource(id: string) {
const next = new Set(selectedIngestionIds);
if (next.has(id)) {
next.delete(id);
} else {
next.add(id);
}
selectedIngestionIds = next;
}
</script>
<form
method="POST"
{action}
class="space-y-5"
use:enhance={() => {
isLoading = true;
return async ({ result, update }) => {
isLoading = false;
if (result.type === 'success') {
onSuccess();
}
await update({ reset: false });
};
}}
>
<!-- Hidden fields for policy id (edit mode), serialized conditions, and ingestion scope -->
{#if policy}
<input type="hidden" name="id" value={policy.id} />
{/if}
<input type="hidden" name="conditions" value={conditionsJson} />
<input type="hidden" name="ingestionScope" value={ingestionScopeValue} />
<!-- isEnabled as hidden field since Switch is not a native input -->
<input type="hidden" name="isEnabled" value={String(isEnabled)} />
<!-- Name -->
<div class="space-y-1.5">
<Label for="rp-name">{$t('app.retention_policies.name')}</Label>
<Input
id="rp-name"
name="name"
bind:value={name}
required
placeholder="e.g. Legal Department 7-Year"
/>
</div>
<!-- Description -->
<div class="space-y-1.5">
<Label for="rp-description">{$t('app.retention_policies.description')}</Label>
<Input
id="rp-description"
name="description"
bind:value={description}
placeholder="Optional description"
/>
</div>
<div class="grid grid-cols-2 gap-4">
<!-- Priority -->
<div class="space-y-1.5">
<Label for="rp-priority">{$t('app.retention_policies.priority')}</Label>
<Input
id="rp-priority"
name="priority"
type="number"
min={1}
bind:value={priority}
required
/>
</div>
<!-- Retention Period -->
<div class="space-y-1.5">
<Label for="rp-days">{$t('app.retention_policies.retention_period_days')}</Label>
<Input
id="rp-days"
name="retentionPeriodDays"
type="number"
min={1}
bind:value={retentionPeriodDays}
required
/>
</div>
</div>
<!-- Action on Expiry (fixed to delete_permanently for Phase 1) -->
<div class="space-y-1.5">
<Label>{$t('app.retention_policies.action_on_expiry')}</Label>
<Input value={$t('app.retention_policies.delete_permanently')} disabled />
</div>
<!-- Enabled toggle — value written to hidden input above -->
<div class="flex items-center gap-3">
<Switch
id="rp-enabled"
checked={isEnabled}
onCheckedChange={(v) => (isEnabled = v)}
/>
<Label for="rp-enabled">{$t('app.retention_policies.active')}</Label>
</div>
<!-- Ingestion Scope -->
{#if ingestionSources.length > 0}
<div class="space-y-2">
<div class="flex items-center gap-2">
<Database class="text-muted-foreground h-4 w-4" />
<Label>{$t('app.retention_policies.ingestion_scope')}</Label>
</div>
<p class="text-muted-foreground text-xs">
{$t('app.retention_policies.ingestion_scope_description')}
</p>
<div class="bg-muted/40 rounded-md border p-3">
<!-- "All sources" option -->
<label class="flex cursor-pointer items-center gap-2.5 py-1">
<input
type="checkbox"
class="h-4 w-4 rounded"
checked={selectedIngestionIds.size === 0}
onchange={() => {
selectedIngestionIds = new Set();
}}
/>
<span class="text-sm font-medium italic">
{$t('app.retention_policies.ingestion_scope_all')}
</span>
</label>
<div class="my-2 border-t"></div>
{#each ingestionSources as source (source.id)}
<label class="flex cursor-pointer items-center gap-2.5 py-1">
<input
type="checkbox"
class="h-4 w-4 rounded"
checked={selectedIngestionIds.has(source.id)}
onchange={() => toggleIngestionSource(source.id)}
/>
<span class="text-sm">{source.name}</span>
<Badge variant="secondary" class="ml-auto text-[10px]">
{source.provider.replace(/_/g, ' ')}
</Badge>
</label>
{/each}
</div>
{#if selectedIngestionIds.size > 0}
<p class="text-muted-foreground text-xs">
{($t as any)('app.retention_policies.ingestion_scope_selected', {
count: selectedIngestionIds.size,
})}
</p>
{/if}
</div>
{/if}
<!-- Conditions builder -->
<div class="space-y-3">
<div class="flex items-center justify-between">
<Label>{$t('app.retention_policies.conditions')}</Label>
{#if rules.length > 1}
<Select.Root
type="single"
value={logicalOperator}
onValueChange={(v) => (logicalOperator = v as LogicalOperator)}
>
<Select.Trigger class="h-8 w-24 text-xs">
{logicalOperator}
</Select.Trigger>
<Select.Content>
<Select.Item value="AND">{$t('app.retention_policies.and')}</Select.Item>
<Select.Item value="OR">{$t('app.retention_policies.or')}</Select.Item>
</Select.Content>
</Select.Root>
{/if}
</div>
<p class="text-muted-foreground text-xs">
{$t('app.retention_policies.conditions_description')}
</p>
{#each rules as rule, i (i)}
<div class="bg-muted/40 flex items-center gap-2 rounded-md border p-3">
<!-- Field selector -->
<Select.Root
type="single"
value={rule.field}
onValueChange={(v) => updateRuleField(i, v as ConditionField)}
>
<Select.Trigger class="h-8 flex-1 text-xs">
{fieldOptions.find((f) => f.value === rule.field)?.label ?? rule.field}
</Select.Trigger>
<Select.Content>
{#each fieldOptions as opt}
<Select.Item value={opt.value}>{opt.label}</Select.Item>
{/each}
</Select.Content>
</Select.Root>
<!-- Operator selector -->
<Select.Root
type="single"
value={rule.operator}
onValueChange={(v) => updateRuleOperator(i, v as ConditionOperator)}
>
<Select.Trigger class="h-8 flex-1 text-xs">
{operatorOptions.find((o) => o.value === rule.operator)?.label ?? rule.operator}
</Select.Trigger>
<Select.Content>
{#each operatorOptions as opt}
<Select.Item value={opt.value}>{opt.label}</Select.Item>
{/each}
</Select.Content>
</Select.Root>
<!-- Value input -->
<Input
class="h-8 flex-1 text-xs"
value={rule.value}
oninput={(e) => updateRuleValue(i, (e.target as HTMLInputElement).value)}
placeholder={$t('app.retention_policies.value_placeholder')}
required
/>
<!-- Remove rule -->
<Button
type="button"
variant="ghost"
size="icon"
class="h-8 w-8 shrink-0"
onclick={() => removeRule(i)}
aria-label={$t('app.retention_policies.remove_rule')}
>
<Trash2 class="h-4 w-4" />
</Button>
</div>
{/each}
<Button type="button" variant="outline" size="sm" onclick={addRule}>
<Plus class="mr-1.5 h-4 w-4" />
{$t('app.retention_policies.add_rule')}
</Button>
</div>
<!-- Actions -->
<div class="flex justify-end gap-2 pt-2">
<Button type="button" variant="outline" onclick={onCancel} disabled={isLoading}>
{$t('app.retention_policies.cancel')}
</Button>
<Button type="submit" disabled={isLoading}>
{#if isLoading}
{$t('app.components.common.submitting')}
{:else if policy}
{$t('app.retention_policies.save')}
{:else}
{$t('app.retention_policies.create')}
{/if}
</Button>
</div>
</form>

99
packages/frontend/src/lib/translations/en.json
View File

@@ -319,6 +319,88 @@
"top_10_senders": "Top 10 Senders",
"no_indexed_insights": "No indexed insights available."
},
"retention_policies": {
"title": "Retention Policies",
"header": "Retention Policies",
"meta_description": "Manage data retention policies to automate email lifecycle and compliance.",
"create_new": "Create New Policy",
"no_policies_found": "No retention policies found.",
"name": "Name",
"description": "Description",
"priority": "Priority",
"retention_period": "Retention Period",
"retention_period_days": "Retention Period (days)",
"action_on_expiry": "Action on Expiry",
"delete_permanently": "Delete Permanently",
"status": "Status",
"active": "Active",
"inactive": "Inactive",
"conditions": "Conditions",
"conditions_description": "Define rules to match emails. If no conditions are set, the policy applies to all emails.",
"logical_operator": "Logical Operator",
"and": "AND",
"or": "OR",
"add_rule": "Add Rule",
"remove_rule": "Remove Rule",
"field": "Field",
"field_sender": "Sender",
"field_recipient": "Recipient",
"field_subject": "Subject",
"field_attachment_type": "Attachment Type",
"operator": "Operator",
"operator_equals": "Equals",
"operator_not_equals": "Not Equals",
"operator_contains": "Contains",
"operator_not_contains": "Not Contains",
"operator_starts_with": "Starts With",
"operator_ends_with": "Ends With",
"operator_domain_match": "Domain Match",
"operator_regex_match": "Regex Match",
"value": "Value",
"value_placeholder": "e.g. user@example.com",
"edit": "Edit",
"delete": "Delete",
"create": "Create",
"save": "Save Changes",
"cancel": "Cancel",
"create_description": "Create a new retention policy to manage the lifecycle of archived emails.",
"edit_description": "Update the settings for this retention policy.",
"delete_confirmation_title": "Delete this retention policy?",
"delete_confirmation_description": "This action cannot be undone. Emails matched by this policy will no longer be subject to automatic deletion.",
"deleting": "Deleting",
"confirm": "Confirm",
"days": "days",
"no_conditions": "All emails (no filter)",
"rules": "rules",
"simulator_title": "Policy Simulator",
"simulator_description": "Test an email's metadata against all active policies to see which retention period would apply.",
"simulator_sender": "Sender Email",
"simulator_sender_placeholder": "e.g. john@finance.company.de",
"simulator_recipients": "Recipients",
"simulator_recipients_placeholder": "Comma-separated, e.g. jane@company.de, bob@company.de",
"simulator_subject": "Subject",
"simulator_subject_placeholder": "e.g. Q4 Tax Report",
"simulator_attachment_types": "Attachment Types",
"simulator_attachment_types_placeholder": "Comma-separated, e.g. .pdf, .xlsx",
"simulator_run": "Run Simulation",
"simulator_running": "Running...",
"simulator_result_title": "Simulation Result",
"simulator_no_match": "No active policy matched this email. It will not be subject to automated deletion.",
"simulator_matched": "Matched — retention period of {{days}} days applies.",
"simulator_matching_policies": "Matching Policy IDs",
"simulator_no_result": "Run a simulation to see which policies apply to a given email.",
"simulator_ingestion_source": "Simulate for Ingestion Source",
"simulator_ingestion_source_description": "Select an ingestion source to test scoped policies. Leave blank to evaluate against all policies regardless of scope.",
"simulator_ingestion_all": "All sources (ignore scope)",
"ingestion_scope": "Ingestion Scope",
"ingestion_scope_description": "Restrict this policy to specific ingestion sources. Leave all unchecked to apply to all sources.",
"ingestion_scope_all": "All ingestion sources",
"ingestion_scope_selected": "{{count}} source(s) selected — this policy will only apply to emails from those sources.",
"create_success": "Retention policy created successfully.",
"update_success": "Retention policy updated successfully.",
"delete_success": "Retention policy deleted successfully.",
"delete_error": "Failed to delete retention policy."
},
"audit_log": {
"title": "Audit Log",
"header": "Audit Log",
@@ -375,20 +457,22 @@
"license_page": {
"title": "Enterprise License Status",
"meta_description": "View the current status of your Open Archiver Enterprise license.",
"revoked_title": "License Revoked",
"revoked_message": "Your license has been revoked by the license administrator. Enterprise features will be disabled {{grace_period}}. Please contact your account manager for assistance.",
"revoked_grace_period": "on {{date}}",
"revoked_immediately": "immediately",
"revoked_title": "License Invalid",
"revoked_message": "Your license has been revoked or your seat overage grace period has expired. All enterprise features are now disabled. Please contact your account manager for assistance.",
"notice_title": "Notice",
"seat_limit_exceeded_title": "Seat Limit Exceeded",
"seat_limit_exceeded_message": "Your license is for {{planSeats}} users, but you are currently using {{activeSeats}}. Please contact sales to adjust your subscription.",
"seat_limit_exceeded_message": "Your license covers {{planSeats}} seats but {{activeSeats}} are currently in use. Please reduce usage or upgrade your plan.",
"seat_limit_grace_deadline": "Enterprise features will be disabled on {{date}} unless the seat count is reduced.",
"customer": "Customer",
"license_details": "License Details",
"license_status": "License Status",
"active": "Active",
"expired": "Expired",
"revoked": "Revoked",
"overage": "Seat Overage",
"unknown": "Unknown",
"expires": "Expires",
"last_checked": "Last verified",
"seat_usage": "Seat Usage",
"seats_used": "{{activeSeats}} of {{planSeats}} seats used",
"enabled_features": "Enabled Features",
@@ -398,7 +482,10 @@
"enabled": "Enabled",
"disabled": "Disabled",
"could_not_load_title": "Could Not Load License",
"could_not_load_message": "An unexpected error occurred."
"could_not_load_message": "An unexpected error occurred.",
"revalidate": "Revalidate License",
"revalidating": "Revalidating...",
"revalidate_success": "License revalidated successfully."
}
}
}

14
packages/frontend/src/routes/dashboard/+layout.svelte
View File

@@ -8,6 +8,7 @@
import { page } from '$app/state';
import ThemeSwitcher from '$lib/components/custom/ThemeSwitcher.svelte';
import { t } from '$lib/translations';
import Badge from '$lib/components/ui/badge/badge.svelte';
let { data, children } = $props();
interface NavItem {
@@ -76,7 +77,13 @@
const enterpriseNavItems: NavItem[] = [
{
label: 'Compliance',
subMenu: [{ href: '/dashboard/compliance/audit-log', label: 'Audit Log' }],
subMenu: [
{ href: '/dashboard/compliance/audit-log', label: $t('app.audit_log.title') },
{
href: '/dashboard/compliance/retention-policies',
label: $t('app.retention_policies.title'),
},
],
position: 3,
},
{
@@ -130,6 +137,9 @@
<a href="/dashboard" class="flex flex-row items-center gap-2 font-bold">
<img src="/logos/logo-sq.svg" alt="OpenArchiver Logo" class="h-8 w-8" />
<span class="hidden sm:inline-block">Open Archiver</span>
{#if data.enterpriseMode}
<Badge class="text-[8px] font-bold px-1 py-0.5">Enterprise</Badge>
{/if}
</a>
<!-- Desktop Navigation -->
@@ -151,7 +161,7 @@
{item.label}
</NavigationMenu.Trigger>
<NavigationMenu.Content>
<ul class="grid w-fit min-w-32 gap-1 p-1">
<ul class="grid w-fit min-w-40 gap-1 p-1">
{#each item.subMenu as subItem}
<li>
<NavigationMenu.Link href={subItem.href}>

184
packages/frontend/src/routes/dashboard/compliance/retention-policies/+page.server.ts Normal file
View File

@@ -0,0 +1,184 @@
import { api } from '$lib/server/api';
import { error } from '@sveltejs/kit';
import type { Actions, PageServerLoad } from './$types';
import type { RetentionPolicy, PolicyEvaluationResult, SafeIngestionSource } from '@open-archiver/types';
export const load: PageServerLoad = async (event) => {
if (!event.locals.enterpriseMode) {
throw error(
403,
'This feature is only available in the Enterprise Edition. Please contact Open Archiver to upgrade.'
);
}
// Fetch policies and ingestion sources in parallel
const [policiesRes, ingestionsRes] = await Promise.all([
api('/enterprise/retention-policy/policies', event),
api('/ingestion-sources', event),
]);
const policiesJson = await policiesRes.json();
if (!policiesRes.ok) {
throw error(policiesRes.status, policiesJson.message || JSON.stringify(policiesJson));
}
// Ingestion sources are best-effort — don't hard-fail if unavailable
let ingestionSources: SafeIngestionSource[] = [];
if (ingestionsRes.ok) {
const ingestionsJson = await ingestionsRes.json();
ingestionSources = Array.isArray(ingestionsJson) ? ingestionsJson : [];
}
const policies: RetentionPolicy[] = policiesJson;
return { policies, ingestionSources };
};
export const actions: Actions = {
create: async (event) => {
const data = await event.request.formData();
const conditionsRaw = JSON.parse(
(data.get('conditions') as string) || '{"logicalOperator":"AND","rules":[]}'
);
// Parse ingestionScope: comma-separated UUIDs, or empty = null (all sources)
const ingestionScopeRaw = (data.get('ingestionScope') as string) || '';
const ingestionScope =
ingestionScopeRaw.trim().length > 0
? ingestionScopeRaw
.split(',')
.map((s) => s.trim())
.filter(Boolean)
: null;
const body = {
name: data.get('name') as string,
description: (data.get('description') as string) || undefined,
priority: Number(data.get('priority')),
retentionPeriodDays: Number(data.get('retentionPeriodDays')),
actionOnExpiry: 'delete_permanently' as const,
isEnabled: data.get('isEnabled') === 'true',
// Send null when no rules — means "apply to all emails"
conditions: conditionsRaw.rules.length > 0 ? conditionsRaw : null,
ingestionScope,
};
const response = await api('/enterprise/retention-policy/policies', event, {
method: 'POST',
body: JSON.stringify(body),
});
const res = await response.json();
if (!response.ok) {
return { success: false, message: res.message || 'Failed to create policy' };
}
return { success: true };
},
update: async (event) => {
const data = await event.request.formData();
const id = data.get('id') as string;
const conditionsRaw = JSON.parse(
(data.get('conditions') as string) || '{"logicalOperator":"AND","rules":[]}'
);
const ingestionScopeRaw = (data.get('ingestionScope') as string) || '';
const ingestionScope =
ingestionScopeRaw.trim().length > 0
? ingestionScopeRaw
.split(',')
.map((s) => s.trim())
.filter(Boolean)
: null;
const body = {
name: data.get('name') as string,
description: (data.get('description') as string) || undefined,
priority: Number(data.get('priority')),
retentionPeriodDays: Number(data.get('retentionPeriodDays')),
actionOnExpiry: 'delete_permanently' as const,
isEnabled: data.get('isEnabled') === 'true',
conditions: conditionsRaw.rules.length > 0 ? conditionsRaw : null,
ingestionScope,
};
const response = await api(`/enterprise/retention-policy/policies/${id}`, event, {
method: 'PUT',
body: JSON.stringify(body),
});
const res = await response.json();
if (!response.ok) {
return { success: false, message: res.message || 'Failed to update policy' };
}
return { success: true };
},
delete: async (event) => {
const data = await event.request.formData();
const id = data.get('id') as string;
const response = await api(`/enterprise/retention-policy/policies/${id}`, event, {
method: 'DELETE',
});
if (!response.ok) {
const res = await response.json().catch(() => ({}));
return { success: false, message: res.message || 'Failed to delete policy' };
}
return { success: true };
},
evaluate: async (event) => {
const data = await event.request.formData();
// Parse recipients and attachment types from comma-separated strings
const recipientsRaw = (data.get('recipients') as string) || '';
const attachmentTypesRaw = (data.get('attachmentTypes') as string) || '';
const ingestionSourceId = (data.get('ingestionSourceId') as string) || undefined;
const body = {
emailMetadata: {
sender: (data.get('sender') as string) || '',
recipients: recipientsRaw
.split(',')
.map((s) => s.trim())
.filter(Boolean),
subject: (data.get('subject') as string) || '',
attachmentTypes: attachmentTypesRaw
.split(',')
.map((s) => s.trim())
.filter(Boolean),
// Only include ingestionSourceId if a non-empty value was provided
...(ingestionSourceId ? { ingestionSourceId } : {}),
},
};
const response = await api('/enterprise/retention-policy/policies/evaluate', event, {
method: 'POST',
body: JSON.stringify(body),
});
const res = await response.json();
if (!response.ok) {
return {
success: false,
message: res.message || 'Failed to evaluate policies',
evaluationResult: null as PolicyEvaluationResult | null,
};
}
return {
success: true,
evaluationResult: res as PolicyEvaluationResult,
};
},
};

460
packages/frontend/src/routes/dashboard/compliance/retention-policies/+page.svelte Normal file
View File

@@ -0,0 +1,460 @@
<script lang="ts">
import type { ActionData, PageData } from './$types';
import { t } from '$lib/translations';
import { Button } from '$lib/components/ui/button';
import { Badge } from '$lib/components/ui/badge';
import * as Table from '$lib/components/ui/table';
import * as Dialog from '$lib/components/ui/dialog/index.js';
import * as DropdownMenu from '$lib/components/ui/dropdown-menu/index.js';
import * as Select from '$lib/components/ui/select/index.js';
import { enhance } from '$app/forms';
import { MoreHorizontal, Plus, FlaskConical } from 'lucide-svelte';
import { toast } from 'svelte-sonner';
import RetentionPolicyForm from '$lib/components/custom/RetentionPolicyForm.svelte';
import { Input } from '$lib/components/ui/input';
import { Label } from '$lib/components/ui/label';
import type { RetentionPolicy, PolicyEvaluationResult } from '@open-archiver/types';
let { data, form }: { data: PageData; form: ActionData } = $props();
let policies = $derived(data.policies);
let ingestionSources = $derived(data.ingestionSources);
// --- Dialog state ---
let isCreateOpen = $state(false);
let isEditOpen = $state(false);
let isDeleteOpen = $state(false);
let selectedPolicy = $state<RetentionPolicy | null>(null);
let isFormLoading = $state(false);
let isDeleting = $state(false);
// --- Simulator state ---
let isSimulating = $state(false);
let evaluationResult = $state<PolicyEvaluationResult | null>(null);
/** The ingestion source ID selected for the simulator (empty string = all sources / no filter) */
let simIngestionSourceId = $state('');
function openEdit(policy: RetentionPolicy) {
selectedPolicy = policy;
isEditOpen = true;
}
function openDelete(policy: RetentionPolicy) {
selectedPolicy = policy;
isDeleteOpen = true;
}
// React to form results (errors and evaluation results)
$effect(() => {
if (form && form.success === false && form.message) {
toast.error(form.message);
}
if (form && 'evaluationResult' in form) {
evaluationResult = form.evaluationResult ?? null;
}
});
/** Returns a human-readable summary of the conditions on a policy. */
function conditionsSummary(policy: RetentionPolicy): string {
if (!policy.conditions || policy.conditions.rules.length === 0) {
return $t('app.retention_policies.no_conditions');
}
const count = policy.conditions.rules.length;
const op = policy.conditions.logicalOperator;
return `${count} ${$t('app.retention_policies.rules')} (${op})`;
}
</script>
<svelte:head>
<title>{$t('app.retention_policies.title')} - Open Archiver</title>
<meta name="description" content={$t('app.retention_policies.meta_description')} />
<meta
name="keywords"
content="retention policies, data retention, email lifecycle, compliance, GDPR"
/>
</svelte:head>
<div class="mb-6 flex items-center justify-between">
<h1 class="text-2xl font-bold">{$t('app.retention_policies.header')}</h1>
<Button onclick={() => (isCreateOpen = true)}>
<Plus class="mr-1.5 h-4 w-4" />
{$t('app.retention_policies.create_new')}
</Button>
</div>
<div class="rounded-md border">
<Table.Root>
<Table.Header>
<Table.Row>
<Table.Head>{$t('app.retention_policies.name')}</Table.Head>
<Table.Head>{$t('app.retention_policies.priority')}</Table.Head>
<Table.Head>{$t('app.retention_policies.retention_period')}</Table.Head>
<Table.Head>{$t('app.retention_policies.ingestion_scope')}</Table.Head>
<Table.Head>{$t('app.retention_policies.conditions')}</Table.Head>
<Table.Head>{$t('app.retention_policies.status')}</Table.Head>
<Table.Head class="text-right">{$t('app.ingestions.actions')}</Table.Head>
</Table.Row>
</Table.Header>
<Table.Body>
{#if policies && policies.length > 0}
{#each policies as policy (policy.id)}
<Table.Row>
<Table.Cell class="font-medium">
<div>{policy.name}</div>
<div class="mt-0.5 font-mono text-[10px] text-muted-foreground">
{policy.id}
</div>
{#if policy.description}
<div class="text-muted-foreground mt-0.5 text-xs">{policy.description}</div>
{/if}
</Table.Cell>
<Table.Cell>{policy.priority}</Table.Cell>
<Table.Cell>
{policy.retentionPeriodDays}
{$t('app.retention_policies.days')}
</Table.Cell>
<Table.Cell>
{#if !policy.ingestionScope || policy.ingestionScope.length === 0}
<span class="text-muted-foreground text-sm italic">
{$t('app.retention_policies.ingestion_scope_all')}
</span>
{:else}
<div class="flex flex-wrap gap-1">
{#each policy.ingestionScope as sourceId (sourceId)}
{@const source = ingestionSources.find((s) => s.id === sourceId)}
<Badge variant="outline" class="text-xs">
{source?.name ?? sourceId.slice(0, 8) + '…'}
</Badge>
{/each}
</div>
{/if}
</Table.Cell>
<Table.Cell>
<span class="text-muted-foreground text-sm">{conditionsSummary(policy)}</span>
</Table.Cell>
<Table.Cell>
{#if policy.isActive}
<Badge variant="default" class="bg-green-500 text-white">
{$t('app.retention_policies.active')}
</Badge>
{:else}
<Badge variant="secondary">
{$t('app.retention_policies.inactive')}
</Badge>
{/if}
</Table.Cell>
<Table.Cell class="text-right">
<DropdownMenu.Root>
<DropdownMenu.Trigger>
{#snippet child({ props })}
<Button
{...props}
variant="ghost"
size="icon"
class="h-8 w-8"
aria-label={$t('app.ingestions.open_menu')}
>
<MoreHorizontal class="h-4 w-4" />
</Button>
{/snippet}
</DropdownMenu.Trigger>
<DropdownMenu.Content align="end">
<DropdownMenu.Item onclick={() => openEdit(policy)}>
{$t('app.retention_policies.edit')}
</DropdownMenu.Item>
<DropdownMenu.Item
class="text-destructive focus:text-destructive"
onclick={() => openDelete(policy)}
>
{$t('app.retention_policies.delete')}
</DropdownMenu.Item>
</DropdownMenu.Content>
</DropdownMenu.Root>
</Table.Cell>
</Table.Row>
{/each}
{:else}
<Table.Row>
<Table.Cell colspan={7} class="h-24 text-center">
{$t('app.retention_policies.no_policies_found')}
</Table.Cell>
</Table.Row>
{/if}
</Table.Body>
</Table.Root>
</div>
<!-- Create dialog -->
<Dialog.Root bind:open={isCreateOpen}>
<Dialog.Content class="sm:max-w-[600px]">
<Dialog.Header>
<Dialog.Title>{$t('app.retention_policies.create')}</Dialog.Title>
<Dialog.Description>
{$t('app.retention_policies.create_description')}
</Dialog.Description>
</Dialog.Header>
<div class="max-h-[70vh] overflow-y-auto pr-1">
<RetentionPolicyForm
action="?/create"
{ingestionSources}
bind:isLoading={isFormLoading}
onCancel={() => (isCreateOpen = false)}
onSuccess={() => {
isCreateOpen = false;
toast.success($t('app.retention_policies.create_success'));
}}
/>
</div>
</Dialog.Content>
</Dialog.Root>
<!-- Edit dialog -->
<Dialog.Root bind:open={isEditOpen}>
<Dialog.Content class="sm:max-w-[600px]">
<Dialog.Header>
<Dialog.Title>{$t('app.retention_policies.edit')}</Dialog.Title>
<Dialog.Description>
{$t('app.retention_policies.edit_description')}
</Dialog.Description>
</Dialog.Header>
{#if selectedPolicy}
<div class="max-h-[70vh] overflow-y-auto pr-1">
<RetentionPolicyForm
policy={selectedPolicy}
action="?/update"
{ingestionSources}
bind:isLoading={isFormLoading}
onCancel={() => (isEditOpen = false)}
onSuccess={() => {
isEditOpen = false;
selectedPolicy = null;
toast.success($t('app.retention_policies.update_success'));
}}
/>
</div>
{/if}
</Dialog.Content>
</Dialog.Root>
<!-- Policy Simulator -->
<div class="mt-8 rounded-md border">
<div class="flex items-center gap-2 border-b px-6 py-4">
<FlaskConical class="text-muted-foreground h-5 w-5" />
<div>
<h2 class="text-base font-semibold">{$t('app.retention_policies.simulator_title')}</h2>
<p class="text-muted-foreground text-sm">
{$t('app.retention_policies.simulator_description')}
</p>
</div>
</div>
<form
method="POST"
action="?/evaluate"
class="grid gap-6 p-6 md:grid-cols-2"
use:enhance={() => {
isSimulating = true;
evaluationResult = null;
return async ({ update }) => {
isSimulating = false;
await update({ reset: false });
};
}}
>
<!-- Hidden field for selected ingestion source -->
<input type="hidden" name="ingestionSourceId" value={simIngestionSourceId} />
<!-- Sender -->
<div class="space-y-1.5">
<Label for="sim-sender">{$t('app.retention_policies.simulator_sender')}</Label>
<Input
id="sim-sender"
name="sender"
type="email"
placeholder={$t('app.retention_policies.simulator_sender_placeholder')}
/>
</div>
<!-- Subject -->
<div class="space-y-1.5">
<Label for="sim-subject">{$t('app.retention_policies.simulator_subject')}</Label>
<Input
id="sim-subject"
name="subject"
placeholder={$t('app.retention_policies.simulator_subject_placeholder')}
/>
</div>
<!-- Recipients -->
<div class="space-y-1.5">
<Label for="sim-recipients">{$t('app.retention_policies.simulator_recipients')}</Label>
<Input
id="sim-recipients"
name="recipients"
placeholder={$t('app.retention_policies.simulator_recipients_placeholder')}
/>
</div>
<!-- Attachment Types -->
<div class="space-y-1.5">
<Label for="sim-attachment-types">
{$t('app.retention_policies.simulator_attachment_types')}
</Label>
<Input
id="sim-attachment-types"
name="attachmentTypes"
placeholder={$t('app.retention_policies.simulator_attachment_types_placeholder')}
/>
</div>
<!-- Ingestion Source filter (only shown when sources are available) -->
{#if ingestionSources.length > 0}
<div class="space-y-1.5 md:col-span-2">
<Label>{$t('app.retention_policies.simulator_ingestion_source')}</Label>
<p class="text-muted-foreground text-xs">
{$t('app.retention_policies.simulator_ingestion_source_description')}
</p>
<Select.Root
type="single"
value={simIngestionSourceId}
onValueChange={(v) => (simIngestionSourceId = v)}
>
<Select.Trigger class="w-full">
{#if simIngestionSourceId}
{ingestionSources.find((s) => s.id === simIngestionSourceId)?.name ??
$t('app.retention_policies.simulator_ingestion_all')}
{:else}
{$t('app.retention_policies.simulator_ingestion_all')}
{/if}
</Select.Trigger>
<Select.Content>
<Select.Item value="">
<span class="italic">{$t('app.retention_policies.simulator_ingestion_all')}</span>
</Select.Item>
{#each ingestionSources as source (source.id)}
<Select.Item value={source.id}>
{source.name}
<span class="text-muted-foreground ml-1 text-xs">
({source.provider.replace(/_/g, ' ')})
</span>
</Select.Item>
{/each}
</Select.Content>
</Select.Root>
</div>
{/if}
<!-- Submit spans full width on md -->
<div class="flex items-end md:col-span-2">
<Button type="submit" disabled={isSimulating} class="w-full md:w-auto">
<FlaskConical class="mr-1.5 h-4 w-4" />
{#if isSimulating}
{$t('app.retention_policies.simulator_running')}
{:else}
{$t('app.retention_policies.simulator_run')}
{/if}
</Button>
</div>
</form>
<!-- Result panel — shown only after a simulation has been run -->
{#if evaluationResult !== null}
<div class="border-t px-6 py-4">
<h3 class="mb-3 text-sm font-semibold">
{$t('app.retention_policies.simulator_result_title')}
</h3>
{#if evaluationResult.appliedRetentionDays === 0}
<div class="bg-muted rounded-md p-4 text-sm">
{$t('app.retention_policies.simulator_no_match')}
</div>
{:else}
<div class="space-y-3">
<div class="rounded-md border border-green-200 bg-green-50 p-4 dark:border-green-800 dark:bg-green-950">
<p class="text-sm font-medium text-green-800 dark:text-green-200">
{($t as any)('app.retention_policies.simulator_matched', {
days: evaluationResult.appliedRetentionDays,
})}
</p>
</div>
{#if evaluationResult.matchingPolicyIds.length > 0}
<div class="space-y-1.5">
<p class="text-muted-foreground text-xs font-medium uppercase tracking-wide">
{$t('app.retention_policies.simulator_matching_policies')}
</p>
<div class="flex flex-wrap gap-2">
{#each evaluationResult.matchingPolicyIds as policyId (policyId)}
{@const matchedPolicy = policies.find((p) => p.id === policyId)}
<div class="flex items-center gap-1.5">
<code class="bg-muted rounded px-2 py-0.5 font-mono text-xs">
{policyId}
</code>
{#if matchedPolicy}
<span class="text-muted-foreground text-xs">({matchedPolicy.name})</span>
{/if}
</div>
{/each}
</div>
</div>
{/if}
</div>
{/if}
</div>
{:else if !isSimulating}
<div class="border-t px-6 py-4">
<p class="text-muted-foreground text-sm">
{$t('app.retention_policies.simulator_no_result')}
</p>
</div>
{/if}
</div>
<!-- Delete confirmation dialog -->
<Dialog.Root bind:open={isDeleteOpen}>
<Dialog.Content>
<Dialog.Header>
<Dialog.Title>{$t('app.retention_policies.delete_confirmation_title')}</Dialog.Title>
<Dialog.Description>
{$t('app.retention_policies.delete_confirmation_description')}
</Dialog.Description>
</Dialog.Header>
<Dialog.Footer>
<Button
variant="outline"
onclick={() => (isDeleteOpen = false)}
disabled={isDeleting}
>
{$t('app.retention_policies.cancel')}
</Button>
{#if selectedPolicy}
<form
method="POST"
action="?/delete"
use:enhance={() => {
isDeleting = true;
return async ({ result, update }) => {
isDeleting = false;
if (result.type === 'success') {
isDeleteOpen = false;
selectedPolicy = null;
toast.success($t('app.retention_policies.delete_success'));
} else {
toast.error($t('app.retention_policies.delete_error'));
}
await update();
};
}}
>
<input type="hidden" name="id" value={selectedPolicy.id} />
<Button type="submit" variant="destructive" disabled={isDeleting}>
{#if isDeleting}
{$t('app.retention_policies.deleting')}
{:else}
{$t('app.retention_policies.confirm')}
{/if}
</Button>
</form>
{/if}
</Dialog.Footer>
</Dialog.Content>
</Dialog.Root>

2
packages/types/src/audit-log.enums.ts
View File

@@ -27,7 +27,9 @@ export const AuditLogTargetTypes = [
'ArchivedEmail',
'Dashboard',
'IngestionSource',
'RetentionPolicy',
'Role',
'SystemEvent',
'SystemSettings',
'User',
'File', // For uploads and downloads

68
packages/types/src/retention.types.ts
View File

@@ -1,11 +1,75 @@
// --- Condition Builder Types ---
export type ConditionField = 'sender' | 'recipient' | 'subject' | 'attachment_type';
/**
* All supported string-matching operators for retention rule conditions.
* - equals / not_equals: exact case-insensitive match
* - contains / not_contains: substring match
* - starts_with: prefix match
* - ends_with: suffix match
* - domain_match: email address ends with @<domain>
* - regex_match: ECMAScript regex (server-side only, length-limited for safety)
*/
export type ConditionOperator =
| 'equals'
| 'not_equals'
| 'contains'
| 'not_contains'
| 'starts_with'
| 'ends_with'
| 'domain_match'
| 'regex_match';
export type LogicalOperator = 'AND' | 'OR';
export interface RetentionRule {
field: ConditionField;
operator: ConditionOperator;
value: string;
}
export interface RetentionRuleGroup {
logicalOperator: LogicalOperator;
rules: RetentionRule[];
}
// --- Policy Evaluation Types ---
export interface PolicyEvaluationRequest {
emailMetadata: {
sender: string;
recipients: string[];
subject: string;
attachmentTypes: string[]; // e.g. ['.pdf', '.xml']
/** Optional ingestion source ID to scope the evaluation. */
ingestionSourceId?: string;
};
}
export interface PolicyEvaluationResult {
appliedRetentionDays: number;
actionOnExpiry: 'delete_permanently';
matchingPolicyIds: string[];
}
// --- Entity Types ---
export interface RetentionPolicy {
id: string;
name: string;
description?: string;
priority: number;
conditions: Record<string, any>; // JSON condition logic
conditions: RetentionRuleGroup | null;
/**
* Restricts the policy to specific ingestion sources.
* null means the policy applies to all ingestion sources.
*/
ingestionScope: string[] | null;
retentionPeriodDays: number;
isActive: boolean;
createdAt: string; // ISO Date string
updatedAt: string; // ISO Date string
}
export interface RetentionLabel {
@@ -21,7 +85,7 @@ export interface RetentionEvent {
eventName: string;
eventType: string; // e.g., 'EMPLOYEE_EXIT'
eventTimestamp: string; // ISO Date string
targetCriteria: Record<string, any>; // JSON criteria
targetCriteria: Record<string, unknown>; // JSON criteria
createdAt: string; // ISO Date string
}