From a7e6b93c778607a2197a1483e842f92a7a01318f Mon Sep 17 00:00:00 2001 From: Wayne <5291640+ringoinca@users.noreply.github.com> Date: Mon, 28 Jul 2025 02:14:38 +0300 Subject: [PATCH] Docs update --- docs/api/archived-email.md | 111 +++++++++++++++++ docs/api/auth.md | 84 +++++++++++++ docs/api/authentication.md | 60 +++++++++ docs/api/dashboard.md | 114 +++++++++++++++++ docs/api/ingestion.md | 248 ++++++++++++++++--------------------- docs/api/search.md | 50 ++++++++ docs/api/storage.md | 26 ++++ docs/index.md | 12 +- 8 files changed, 556 insertions(+), 149 deletions(-) create mode 100644 docs/api/archived-email.md create mode 100644 docs/api/auth.md create mode 100644 docs/api/authentication.md create mode 100644 docs/api/dashboard.md create mode 100644 docs/api/search.md create mode 100644 docs/api/storage.md diff --git a/docs/api/archived-email.md b/docs/api/archived-email.md new file mode 100644 index 0000000..625581f --- /dev/null +++ b/docs/api/archived-email.md @@ -0,0 +1,111 @@ +# Archived Email Service API + +The Archived Email Service is responsible for retrieving archived emails and their details from the database and storage. + +## Endpoints + +All endpoints in this service require authentication. + +### GET /api/v1/archived-emails/ingestion-source/:ingestionSourceId + +Retrieves a paginated list of archived emails for a specific ingestion source. + +**Access:** Authenticated + +#### URL Parameters + +| Parameter | Type | Description | +| :------------------ | :----- | :------------------------------------------------ | +| `ingestionSourceId` | string | The ID of the ingestion source to get emails for. | + +#### Query Parameters + +| Parameter | Type | Description | Default | +| :-------- | :----- | :------------------------------ | :------ | +| `page` | number | The page number for pagination. | 1 | +| `limit` | number | The number of items per page. | 10 | + +#### Responses + +- **200 OK:** A paginated list of archived emails. + + ```json + { + "items": [ + { + "id": "email-id", + "subject": "Test Email", + "from": "sender@example.com", + "sentAt": "2023-10-27T10:00:00.000Z", + "hasAttachments": true, + "recipients": [ + { "name": "Recipient 1", "email": "recipient1@example.com" } + ] + } + ], + "total": 100, + "page": 1, + "limit": 10 + } + ``` + +- **500 Internal Server Error:** An unexpected error occurred. + +### GET /api/v1/archived-emails/:id + +Retrieves a single archived email by its ID, including its raw content and attachments. + +**Access:** Authenticated + +#### URL Parameters + +| Parameter | Type | Description | +| :-------- | :----- | :---------------------------- | +| `id` | string | The ID of the archived email. | + +#### Responses + +- **200 OK:** The archived email details. + + ```json + { + "id": "email-id", + "subject": "Test Email", + "from": "sender@example.com", + "sentAt": "2023-10-27T10:00:00.000Z", + "hasAttachments": true, + "recipients": [ + { "name": "Recipient 1", "email": "recipient1@example.com" } + ], + "raw": "...", + "attachments": [ + { + "id": "attachment-id", + "filename": "document.pdf", + "mimeType": "application/pdf", + "sizeBytes": 12345 + } + ] + } + ``` + +- **404 Not Found:** The archived email with the specified ID was not found. +- **500 Internal Server Error:** An unexpected error occurred. + +## Service Methods + +### `getArchivedEmails(ingestionSourceId: string, page: number, limit: number): Promise` + +Retrieves a paginated list of archived emails from the database for a given ingestion source. + +- **ingestionSourceId:** The ID of the ingestion source. +- **page:** The page number for pagination. +- **limit:** The number of items per page. +- **Returns:** A promise that resolves to a `PaginatedArchivedEmails` object. + +### `getArchivedEmailById(emailId: string): Promise` + +Retrieves a single archived email by its ID, including its raw content and attachments. + +- **emailId:** The ID of the archived email. +- **Returns:** A promise that resolves to an `ArchivedEmail` object or `null` if not found. diff --git a/docs/api/auth.md b/docs/api/auth.md new file mode 100644 index 0000000..f211d4a --- /dev/null +++ b/docs/api/auth.md @@ -0,0 +1,84 @@ +# Auth Service API + +The Auth Service is responsible for handling user authentication, including login and token verification. + +## Endpoints + +### POST /api/v1/auth/login + +Authenticates a user and returns a JWT if the credentials are valid. + +**Access:** Public + +**Rate Limiting:** This endpoint is rate-limited to prevent brute-force attacks. + +#### Request Body + +| Field | Type | Description | +| :--------- | :----- | :------------------------ | +| `email` | string | The user's email address. | +| `password` | string | The user's password. | + +#### Responses + +- **200 OK:** Authentication successful. + + ```json + { + "accessToken": "your.jwt.token", + "user": { + "id": "user-id", + "email": "user@example.com", + "role": "user" + } + } + ``` + +- **400 Bad Request:** Email or password not provided. + + ```json + { + "message": "Email and password are required" + } + ``` + +- **401 Unauthorized:** Invalid credentials. + + ```json + { + "message": "Invalid credentials" + } + ``` + +- **500 Internal Server Error:** An unexpected error occurred. + + ```json + { + "message": "An internal server error occurred" + } + ``` + +## Service Methods + +### `verifyPassword(password: string, hash: string): Promise` + +Compares a plain-text password with a hashed password to verify its correctness. + +- **password:** The plain-text password. +- **hash:** The hashed password to compare against. +- **Returns:** A promise that resolves to `true` if the password is valid, otherwise `false`. + +### `login(email: string, password: string): Promise` + +Handles the user login process. It finds the user by email, verifies the password, and generates a JWT upon successful authentication. + +- **email:** The user's email. +- **password:** The user's password. +- **Returns:** A promise that resolves to a `LoginResponse` object containing the `accessToken` and `user` details, or `null` if authentication fails. + +### `verifyToken(token: string): Promise` + +Verifies the authenticity and expiration of a JWT. + +- **token:** The JWT string to verify. +- **Returns:** A promise that resolves to the token's `AuthTokenPayload` if valid, otherwise `null`. diff --git a/docs/api/authentication.md b/docs/api/authentication.md new file mode 100644 index 0000000..b284a2c --- /dev/null +++ b/docs/api/authentication.md @@ -0,0 +1,60 @@ +# API Authentication + +To access protected API endpoints, you need to include a JSON Web Token (JWT) in the `Authorization` header of your requests. + +## Obtaining a JWT + +First, you need to authenticate with the `/api/v1/auth/login` endpoint by providing your email and password. If the credentials are correct, the API will return an `accessToken`. + +**Request:** + +```http +POST /api/v1/auth/login +Content-Type: application/json + +{ + "email": "user@example.com", + "password": "your-password" +} +``` + +**Successful Response:** + +```json +{ + "accessToken": "your.jwt.token", + "user": { + "id": "user-id", + "email": "user@example.com", + "role": "user" + } +} +``` + +## Making Authenticated Requests + +Once you have the `accessToken`, you must include it in the `Authorization` header of all subsequent requests to protected endpoints, using the `Bearer` scheme. + +**Example:** + +```http +GET /api/v1/dashboard/stats +Authorization: Bearer your.jwt.token +``` + +If the token is missing, expired, or invalid, the API will respond with a `401 Unauthorized` status code. + +## Using a Super API Key + +Alternatively, for server-to-server communication or scripts, you can use a super API key. This key provides unrestricted access to the API and should be kept secret. + +You can set the `SUPER_API_KEY` in your `.env` file. + +To authenticate using the super API key, include it in the `Authorization` header as a Bearer token. + +**Example:** + +```http +GET /api/v1/dashboard/stats +Authorization: Bearer your-super-secret-api-key +``` diff --git a/docs/api/dashboard.md b/docs/api/dashboard.md new file mode 100644 index 0000000..7951783 --- /dev/null +++ b/docs/api/dashboard.md @@ -0,0 +1,114 @@ +# Dashboard Service API + +The Dashboard Service provides endpoints for retrieving statistics and data for the main dashboard. + +## Endpoints + +All endpoints in this service require authentication. + +### GET /api/v1/dashboard/stats + +Retrieves overall statistics, including the total number of archived emails, total storage used, and the number of failed ingestions in the last 7 days. + +**Access:** Authenticated + +#### Responses + +- **200 OK:** An object containing the dashboard statistics. + + ```json + { + "totalEmailsArchived": 12345, + "totalStorageUsed": 54321098, + "failedIngestionsLast7Days": 3 + } + ``` + +### GET /api/v1/dashboard/ingestion-history + +Retrieves the email ingestion history for the last 30 days, grouped by day. + +**Access:** Authenticated + +#### Responses + +- **200 OK:** An object containing the ingestion history. + + ```json + { + "history": [ + { + "date": "2023-09-27T00:00:00.000Z", + "count": 150 + }, + { + "date": "2023-09-28T00:00:00.000Z", + "count": 200 + } + ] + } + ``` + +### GET /api/v1/dashboard/ingestion-sources + +Retrieves a list of all ingestion sources along with their status and storage usage. + +**Access:** Authenticated + +#### Responses + +- **200 OK:** An array of ingestion source objects. + + ```json + [ + { + "id": "source-id-1", + "name": "Google Workspace", + "provider": "google", + "status": "active", + "storageUsed": 12345678 + }, + { + "id": "source-id-2", + "name": "Microsoft 365", + "provider": "microsoft", + "status": "error", + "storageUsed": 87654321 + } + ] + ``` + +### GET /api/v1/dashboard/recent-syncs + +Retrieves a list of recent synchronization jobs. (Note: This is currently a placeholder and will return an empty array). + +**Access:** Authenticated + +#### Responses + +- **200 OK:** An empty array. + + ```json + [] + ``` + +### GET /api/v1/dashboard/indexed-insights + +Retrieves insights from the indexed email data, such as the top senders. + +**Access:** Authenticated + +#### Responses + +- **200 OK:** An object containing indexed insights. + + ```json + { + "topSenders": [ + { + "sender": "user@example.com", + "count": 42 + } + ] + } + ``` diff --git a/docs/api/ingestion.md b/docs/api/ingestion.md index 1046945..9b81f67 100644 --- a/docs/api/ingestion.md +++ b/docs/api/ingestion.md @@ -1,206 +1,168 @@ -# Ingestion Sources API Documentation +# Ingestion Service API -A guide to using the Ingestion Sources API. +The Ingestion Service manages ingestion sources, which are configurations for connecting to email providers and importing emails. -**Base Path:** `/v1/ingestion-sources` +## Endpoints ---- +All endpoints in this service require authentication. -## Authentication +### POST /api/v1/ingestion -All endpoints in this API are protected and require authentication. Requests must include an `Authorization` header containing a valid Bearer token. This can be a JWT obtained from the login endpoint or a `SUPER_API_KEY` for administrative tasks. +Creates a new ingestion source. -**Header Example:** -`Authorization: Bearer ` +**Access:** Authenticated ---- +#### Request Body -## Core Concepts +The request body should be a `CreateIngestionSourceDto` object. -### Ingestion Providers - -The `provider` field determines the type of email source. Each provider requires a different configuration object, for example: - -- `google_workspace`: For connecting to Google Workspace accounts via OAuth 2.0. -- `microsoft_365`: For connecting to Microsoft 365 accounts via OAuth 2.0. -- `generic_imap`: For connecting to any email server that supports IMAP. - -### Ingestion Status - -The `status` field tracks the state of the ingestion source. - -- `pending_auth`: The source has been created but requires user authorization (OAuth flow). -- `active`: The source is authenticated and ready to sync. -- `syncing`: An import job is currently in progress. -- `importing`: initial syncing in progress -- `paused`: The source is temporarily disabled. -- `error`: An error occurred during the last sync. - ---- - -## 1. Create Ingestion Source - -- **Method:** `POST` -- **Path:** `/` -- **Description:** Registers a new source for email ingestion. The `providerConfig` will vary based on the selected `provider`. - -#### Request Body (`CreateIngestionSourceDto`) - -- `name` (string, required): A user-friendly name for the source (e.g., "Marketing Department G-Suite"). -- `provider` (string, required): One of `google_workspace`, `microsoft_365`, or `generic_imap`. -- `providerConfig` (object, required): Configuration specific to the provider. - -##### `providerConfig` for `google_workspace` / `microsoft_365` - -```json -{ - "name": "Corporate Google Workspace", - "provider": "google_workspace", - "providerConfig": { - "clientId": "your-oauth-client-id.apps.googleusercontent.com", - "clientSecret": "your-super-secret-client-secret", - "redirectUri": "https://yourapp.com/oauth/google/callback" - } -} -``` - -##### `providerConfig` for `generic_imap` - -```json -{ - "name": "Legacy IMAP Server", - "provider": "generic_imap", - "providerConfig": { - "host": "imap.example.com", - "port": 993, - "secure": true, - "username": "archive-user", - "password": "imap-password" - } +```typescript +interface CreateIngestionSourceDto { + name: string; + provider: 'google' | 'microsoft' | 'generic_imap'; + providerConfig: IngestionCredentials; } ``` #### Responses -- **Success (`201 Created`):** Returns the full `IngestionSource` object, which now includes a system-generated `id` and default status. +- **201 Created:** The newly created ingestion source. +- **500 Internal Server Error:** An unexpected error occurred. - ```json - { - "id": "a1b2c3d4-e5f6-7890-1234-567890abcdef", - "name": "Corporate Google Workspace", - "provider": "google_workspace", - "status": "pending_auth", - "createdAt": "2025-07-11T12:00:00.000Z", - "updatedAt": "2025-07-11T12:00:00.000Z", - "providerConfig": { ... } - } - ``` +### GET /api/v1/ingestion -- **Error (`500 Internal Server Error`):** Indicates a server-side problem during creation. +Retrieves all ingestion sources. ---- - -## 2. Get All Ingestion Sources - -- **Method:** `GET` -- **Path:** `/` -- **Description:** Retrieves a list of all configured ingestion sources for the organization. +**Access:** Authenticated #### Responses -- **Success (`200 OK`):** Returns an array of `IngestionSource` objects. +- **200 OK:** An array of ingestion source objects. +- **500 Internal Server Error:** An unexpected error occurred. -- **Error (`500 Internal Server Error`):** Indicates a server-side problem. +### GET /api/v1/ingestion/:id ---- +Retrieves a single ingestion source by its ID. -## 3. Get Ingestion Source by ID - -- **Method:** `GET` -- **Path:** `/:id` -- **Description:** Fetches the details of a specific ingestion source. +**Access:** Authenticated #### URL Parameters -- `id` (string, required): The UUID of the ingestion source. +| Parameter | Type | Description | +| :-------- | :----- | :------------------------------ | +| `id` | string | The ID of the ingestion source. | #### Responses -- **Success (`200 OK`):** Returns the corresponding `IngestionSource` object. +- **200 OK:** The ingestion source object. +- **404 Not Found:** Ingestion source not found. +- **500 Internal Server Error:** An unexpected error occurred. -- **Error (`404 Not Found`):** Returned if no source with the given ID exists. -- **Error (`500 Internal Server Error`):** Indicates a server-side problem. +### PUT /api/v1/ingestion/:id ---- +Updates an existing ingestion source. -## 4. Update Ingestion Source - -- **Method:** `PUT` -- **Path:** `/:id` -- **Description:** Modifies an existing ingestion source. This is useful for changing the name, pausing a source, or updating its configuration. +**Access:** Authenticated #### URL Parameters -- `id` (string, required): The UUID of the ingestion source to update. +| Parameter | Type | Description | +| :-------- | :----- | :------------------------------ | +| `id` | string | The ID of the ingestion source. | -#### Request Body (`UpdateIngestionSourceDto`) +#### Request Body -All fields are optional. Only include the fields you want to change. +The request body should be an `UpdateIngestionSourceDto` object. -```json -{ - "name": "Marketing Dept G-Suite (Paused)", - "status": "paused" +```typescript +interface UpdateIngestionSourceDto { + name?: string; + provider?: 'google' | 'microsoft' | 'generic_imap'; + providerConfig?: IngestionCredentials; + status?: + | 'pending_auth' + | 'auth_success' + | 'importing' + | 'active' + | 'paused' + | 'error'; } ``` #### Responses -- **Success (`200 OK`):** Returns the complete, updated `IngestionSource` object. +- **200 OK:** The updated ingestion source object. +- **404 Not Found:** Ingestion source not found. +- **500 Internal Server Error:** An unexpected error occurred. -- **Error (`404 Not Found`):** Returned if no source with the given ID exists. -- **Error (`500 Internal Server Error`):** Indicates a server-side problem. +### DELETE /api/v1/ingestion/:id ---- +Deletes an ingestion source and all associated data. -## 5. Delete Ingestion Source - -- **Method:** `DELETE` -- **Path:** `/:id` -- **Description:** Permanently removes an ingestion source. This action cannot be undone. +**Access:** Authenticated #### URL Parameters -- `id` (string, required): The UUID of the ingestion source to delete. +| Parameter | Type | Description | +| :-------- | :----- | :------------------------------ | +| `id` | string | The ID of the ingestion source. | #### Responses -- **Success (`204 No Content`):** Indicates successful deletion with no body content. +- **204 No Content:** The ingestion source was deleted successfully. +- **404 Not Found:** Ingestion source not found. +- **500 Internal Server Error:** An unexpected error occurred. -- **Error (`404 Not Found`):** Returned if no source with the given ID exists. -- **Error (`500 Internal Server Error`):** Indicates a server-side problem. +### POST /api/v1/ingestion/:id/import ---- +Triggers the initial import process for an ingestion source. -## 6. Trigger Initial Import - -- **Method:** `POST` -- **Path:** `/:id/sync` -- **Description:** Initiates the email import process for a given source. This is an asynchronous operation that enqueues a background job and immediately returns a response. The status of the source will be updated to `importing`. +**Access:** Authenticated #### URL Parameters -- `id` (string, required): The UUID of the ingestion source to sync. +| Parameter | Type | Description | +| :-------- | :----- | :------------------------------ | +| `id` | string | The ID of the ingestion source. | #### Responses -- **Success (`202 Accepted`):** Confirms that the sync request has been accepted for processing. +- **202 Accepted:** The initial import was triggered successfully. +- **404 Not Found:** Ingestion source not found. +- **500 Internal Server Error:** An unexpected error occurred. - ```json - { - "message": "Initial import triggered successfully." - } - ``` +### POST /api/v1/ingestion/:id/pause -- **Error (`404 Not Found`):** Returned if no source with the given ID exists. -- **Error (`500 Internal Server Error`):** Indicates a server-side problem. +Pauses an active ingestion source. + +**Access:** Authenticated + +#### URL Parameters + +| Parameter | Type | Description | +| :-------- | :----- | :------------------------------ | +| `id` | string | The ID of the ingestion source. | + +#### Responses + +- **200 OK:** The updated ingestion source object with a `paused` status. +- **404 Not Found:** Ingestion source not found. +- **500 Internal Server Error:** An unexpected error occurred. + +### POST /api/v1/ingestion/:id/sync + +Triggers a forced synchronization for an ingestion source. + +**Access:** Authenticated + +#### URL Parameters + +| Parameter | Type | Description | +| :-------- | :----- | :------------------------------ | +| `id` | string | The ID of the ingestion source. | + +#### Responses + +- **202 Accepted:** The force sync was triggered successfully. +- **404 Not Found:** Ingestion source not found. +- **500 Internal Server Error:** An unexpected error occurred. diff --git a/docs/api/search.md b/docs/api/search.md new file mode 100644 index 0000000..48882f5 --- /dev/null +++ b/docs/api/search.md @@ -0,0 +1,50 @@ +# Search Service API + +The Search Service provides an endpoint for searching indexed emails. + +## Endpoints + +All endpoints in this service require authentication. + +### GET /api/v1/search + +Performs a search query against the indexed emails. + +**Access:** Authenticated + +#### Query Parameters + +| Parameter | Type | Description | Default | +| :----------------- | :----- | :--------------------------------------------------------------------- | :------ | +| `keywords` | string | The search query. | | +| `page` | number | The page number for pagination. | 1 | +| `limit` | number | The number of items per page. | 10 | +| `matchingStrategy` | string | The matching strategy to use (`all` or `last`). | `last` | +| `filters` | object | Key-value pairs for filtering results (e.g., `from=user@example.com`). | | + +#### Responses + +- **200 OK:** A search result object. + + ```json + { + "hits": [ + { + "id": "email-id", + "subject": "Test Email", + "from": "sender@example.com", + "_formatted": { + "subject": "Test Email" + } + } + ], + "total": 1, + "page": 1, + "limit": 10, + "totalPages": 1, + "processingTimeMs": 5 + } + ``` + +- **400 Bad Request:** Keywords are required. +- **500 Internal Server Error:** An unexpected error occurred. diff --git a/docs/api/storage.md b/docs/api/storage.md new file mode 100644 index 0000000..b6b3e01 --- /dev/null +++ b/docs/api/storage.md @@ -0,0 +1,26 @@ +# Storage Service API + +The Storage Service provides an endpoint for downloading files from the configured storage provider. + +## Endpoints + +All endpoints in this service require authentication. + +### GET /api/v1/storage/download + +Downloads a file from the storage. + +**Access:** Authenticated + +#### Query Parameters + +| Parameter | Type | Description | +| :-------- | :----- | :------------------------------------------------ | +| `path` | string | The path to the file within the storage provider. | + +#### Responses + +- **200 OK:** The file stream. +- **400 Bad Request:** File path is required or invalid. +- **404 Not Found:** File not found. +- **500 Internal Server Error:** An unexpected error occurred. diff --git a/docs/index.md b/docs/index.md index 87dc4a9..78e5bcc 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,14 +1,14 @@ -# Get Started +# Get Started 👋 Welcome to Open Archiver! This guide will help you get started with setting up and using the platform. -## What is Open Archiver? +## What is Open Archiver? 🛡️ **A secure, sovereign, and affordable open-source platform for email archiving and eDiscovery.** Open Archiver provides a robust, self-hosted solution for archiving, storing, indexing, and searching emails from major platforms, including Google Workspace (Gmail), Microsoft 365, as well as generic IMAP-enabled email inboxes. Use Open Archiver to keep a permanent, tamper-proof record of your communication history, free from vendor lock-in. -## Key Features +## Key Features ✨ - **Universal Ingestion**: Connect to Google Workspace, Microsoft 365, and standard IMAP servers to perform initial bulk imports and maintain continuous, real-time synchronization. - **Secure & Efficient Storage**: Emails are stored in the standard `.eml` format. The system uses deduplication and compression to minimize storage costs. All data is encrypted at rest. @@ -17,13 +17,13 @@ Open Archiver provides a robust, self-hosted solution for archiving, storing, in - **Compliance & Retention**: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD). - **Comprehensive Auditing**: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when (TBD). -## Installation +## Installation 🚀 To get your own instance of Open Archiver running, follow our detailed installation guide: - [Installation Guide](./user-guides/installation.md) -## Data Source Configuration +## Data Source Configuration 🔌 After deploying the application, you will need to configure one or more ingestion sources to begin archiving emails. Follow our detailed guides to connect to your email provider: @@ -31,7 +31,7 @@ After deploying the application, you will need to configure one or more ingestio - [Connecting to Microsoft 365](./user-guides/email-providers/microsoft-365.md) - [Connecting to a Generic IMAP Server](./user-guides/email-providers/imap.md) -## Contributing +## Contributing ❤️ We welcome contributions from the community!