github/OpenArchiver
1
1
Fork 0
mirror of https://github.com/LogicLabs-OU/OpenArchiver.git synced 2026-04-06 00:31:57 +02:00
Code Issues 161 Packages Projects Releases Wiki Activity

Docs update

Browse Source
This commit is contained in:
Wayne
2025-07-28 02:14:38 +03:00
parent 9d3e6fc22e
commit a7e6b93c77
8 changed files with 556 additions and 149 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
docs/api/archived-email.md Normal file
View File

@@ -0,0 +1,111 @@
# Archived Email Service API
The Archived Email Service is responsible for retrieving archived emails and their details from the database and storage.
## Endpoints
All endpoints in this service require authentication.
### GET /api/v1/archived-emails/ingestion-source/:ingestionSourceId
Retrieves a paginated list of archived emails for a specific ingestion source.
**Access:** Authenticated
#### URL Parameters
| Parameter | Type | Description |
| :------------------ | :----- | :------------------------------------------------ |
| `ingestionSourceId` | string | The ID of the ingestion source to get emails for. |
#### Query Parameters
| Parameter | Type | Description | Default |
| :-------- | :----- | :------------------------------ | :------ |
| `page` | number | The page number for pagination. | 1 |
| `limit` | number | The number of items per page. | 10 |
#### Responses
- **200 OK:** A paginated list of archived emails.
```json
{
"items": [
{
"id": "email-id",
"subject": "Test Email",
"from": "sender@example.com",
"sentAt": "2023-10-27T10:00:00.000Z",
"hasAttachments": true,
"recipients": [
{ "name": "Recipient 1", "email": "recipient1@example.com" }
]
}
],
"total": 100,
"page": 1,
"limit": 10
}
```
- **500 Internal Server Error:** An unexpected error occurred.
### GET /api/v1/archived-emails/:id
Retrieves a single archived email by its ID, including its raw content and attachments.
**Access:** Authenticated
#### URL Parameters
| Parameter | Type | Description |
| :-------- | :----- | :---------------------------- |
| `id` | string | The ID of the archived email. |
#### Responses
- **200 OK:** The archived email details.
```json
{
"id": "email-id",
"subject": "Test Email",
"from": "sender@example.com",
"sentAt": "2023-10-27T10:00:00.000Z",
"hasAttachments": true,
"recipients": [
{ "name": "Recipient 1", "email": "recipient1@example.com" }
],
"raw": "...",
"attachments": [
{
"id": "attachment-id",
"filename": "document.pdf",
"mimeType": "application/pdf",
"sizeBytes": 12345
}
]
}
```
- **404 Not Found:** The archived email with the specified ID was not found.
- **500 Internal Server Error:** An unexpected error occurred.
## Service Methods
### `getArchivedEmails(ingestionSourceId: string, page: number, limit: number): Promise<PaginatedArchivedEmails>`
Retrieves a paginated list of archived emails from the database for a given ingestion source.
- **ingestionSourceId:** The ID of the ingestion source.
- **page:** The page number for pagination.
- **limit:** The number of items per page.
- **Returns:** A promise that resolves to a `PaginatedArchivedEmails` object.
### `getArchivedEmailById(emailId: string): Promise<ArchivedEmail | null>`
Retrieves a single archived email by its ID, including its raw content and attachments.
- **emailId:** The ID of the archived email.
- **Returns:** A promise that resolves to an `ArchivedEmail` object or `null` if not found.

84
docs/api/auth.md Normal file
View File

@@ -0,0 +1,84 @@
# Auth Service API
The Auth Service is responsible for handling user authentication, including login and token verification.
## Endpoints
### POST /api/v1/auth/login
Authenticates a user and returns a JWT if the credentials are valid.
**Access:** Public
**Rate Limiting:** This endpoint is rate-limited to prevent brute-force attacks.
#### Request Body
| Field | Type | Description |
| :--------- | :----- | :------------------------ |
| `email` | string | The user's email address. |
| `password` | string | The user's password. |
#### Responses
- **200 OK:** Authentication successful.
```json
{
"accessToken": "your.jwt.token",
"user": {
"id": "user-id",
"email": "user@example.com",
"role": "user"
}
}
```
- **400 Bad Request:** Email or password not provided.
```json
{
"message": "Email and password are required"
}
```
- **401 Unauthorized:** Invalid credentials.
```json
{
"message": "Invalid credentials"
}
```
- **500 Internal Server Error:** An unexpected error occurred.
```json
{
"message": "An internal server error occurred"
}
```
## Service Methods
### `verifyPassword(password: string, hash: string): Promise<boolean>`
Compares a plain-text password with a hashed password to verify its correctness.
- **password:** The plain-text password.
- **hash:** The hashed password to compare against.
- **Returns:** A promise that resolves to `true` if the password is valid, otherwise `false`.
### `login(email: string, password: string): Promise<LoginResponse | null>`
Handles the user login process. It finds the user by email, verifies the password, and generates a JWT upon successful authentication.
- **email:** The user's email.
- **password:** The user's password.
- **Returns:** A promise that resolves to a `LoginResponse` object containing the `accessToken` and `user` details, or `null` if authentication fails.
### `verifyToken(token: string): Promise<AuthTokenPayload | null>`
Verifies the authenticity and expiration of a JWT.
- **token:** The JWT string to verify.
- **Returns:** A promise that resolves to the token's `AuthTokenPayload` if valid, otherwise `null`.

60
docs/api/authentication.md Normal file
View File

@@ -0,0 +1,60 @@
# API Authentication
To access protected API endpoints, you need to include a JSON Web Token (JWT) in the `Authorization` header of your requests.
## Obtaining a JWT
First, you need to authenticate with the `/api/v1/auth/login` endpoint by providing your email and password. If the credentials are correct, the API will return an `accessToken`.
**Request:**
```http
POST /api/v1/auth/login
Content-Type: application/json
{
"email": "user@example.com",
"password": "your-password"
}
```
**Successful Response:**
```json
{
"accessToken": "your.jwt.token",
"user": {
"id": "user-id",
"email": "user@example.com",
"role": "user"
}
}
```
## Making Authenticated Requests
Once you have the `accessToken`, you must include it in the `Authorization` header of all subsequent requests to protected endpoints, using the `Bearer` scheme.
**Example:**
```http
GET /api/v1/dashboard/stats
Authorization: Bearer your.jwt.token
```
If the token is missing, expired, or invalid, the API will respond with a `401 Unauthorized` status code.
## Using a Super API Key
Alternatively, for server-to-server communication or scripts, you can use a super API key. This key provides unrestricted access to the API and should be kept secret.
You can set the `SUPER_API_KEY` in your `.env` file.
To authenticate using the super API key, include it in the `Authorization` header as a Bearer token.
**Example:**
```http
GET /api/v1/dashboard/stats
Authorization: Bearer your-super-secret-api-key
```

114
docs/api/dashboard.md Normal file
View File

@@ -0,0 +1,114 @@
# Dashboard Service API
The Dashboard Service provides endpoints for retrieving statistics and data for the main dashboard.
## Endpoints
All endpoints in this service require authentication.
### GET /api/v1/dashboard/stats
Retrieves overall statistics, including the total number of archived emails, total storage used, and the number of failed ingestions in the last 7 days.
**Access:** Authenticated
#### Responses
- **200 OK:** An object containing the dashboard statistics.
```json
{
"totalEmailsArchived": 12345,
"totalStorageUsed": 54321098,
"failedIngestionsLast7Days": 3
}
```
### GET /api/v1/dashboard/ingestion-history
Retrieves the email ingestion history for the last 30 days, grouped by day.
**Access:** Authenticated
#### Responses
- **200 OK:** An object containing the ingestion history.
```json
{
"history": [
{
"date": "2023-09-27T00:00:00.000Z",
"count": 150
},
{
"date": "2023-09-28T00:00:00.000Z",
"count": 200
}
]
}
```
### GET /api/v1/dashboard/ingestion-sources
Retrieves a list of all ingestion sources along with their status and storage usage.
**Access:** Authenticated
#### Responses
- **200 OK:** An array of ingestion source objects.
```json
[
{
"id": "source-id-1",
"name": "Google Workspace",
"provider": "google",
"status": "active",
"storageUsed": 12345678
},
{
"id": "source-id-2",
"name": "Microsoft 365",
"provider": "microsoft",
"status": "error",
"storageUsed": 87654321
}
]
```
### GET /api/v1/dashboard/recent-syncs
Retrieves a list of recent synchronization jobs. (Note: This is currently a placeholder and will return an empty array).
**Access:** Authenticated
#### Responses
- **200 OK:** An empty array.
```json
[]
```
### GET /api/v1/dashboard/indexed-insights
Retrieves insights from the indexed email data, such as the top senders.
**Access:** Authenticated
#### Responses
- **200 OK:** An object containing indexed insights.
```json
{
"topSenders": [
{
"sender": "user@example.com",
"count": 42
}
]
}
```

248
docs/api/ingestion.md
View File

@@ -1,206 +1,168 @@
# Ingestion Sources API Documentation
# Ingestion Service API
A guide to using the Ingestion Sources API.
The Ingestion Service manages ingestion sources, which are configurations for connecting to email providers and importing emails.
**Base Path:** `/v1/ingestion-sources`
## Endpoints
---
All endpoints in this service require authentication.
## Authentication
### POST /api/v1/ingestion
All endpoints in this API are protected and require authentication. Requests must include an `Authorization` header containing a valid Bearer token. This can be a JWT obtained from the login endpoint or a `SUPER_API_KEY` for administrative tasks.
Creates a new ingestion source.
**Header Example:**
`Authorization: Bearer <YOUR_JWT_OR_SUPER_API_KEY>`
**Access:** Authenticated
---
#### Request Body
## Core Concepts
The request body should be a `CreateIngestionSourceDto` object.
### Ingestion Providers
The `provider` field determines the type of email source. Each provider requires a different configuration object, for example:
- `google_workspace`: For connecting to Google Workspace accounts via OAuth 2.0.
- `microsoft_365`: For connecting to Microsoft 365 accounts via OAuth 2.0.
- `generic_imap`: For connecting to any email server that supports IMAP.
### Ingestion Status
The `status` field tracks the state of the ingestion source.
- `pending_auth`: The source has been created but requires user authorization (OAuth flow).
- `active`: The source is authenticated and ready to sync.
- `syncing`: An import job is currently in progress.
- `importing`: initial syncing in progress
- `paused`: The source is temporarily disabled.
- `error`: An error occurred during the last sync.
---
## 1. Create Ingestion Source
- **Method:** `POST`
- **Path:** `/`
- **Description:** Registers a new source for email ingestion. The `providerConfig` will vary based on the selected `provider`.
#### Request Body (`CreateIngestionSourceDto`)
- `name` (string, required): A user-friendly name for the source (e.g., "Marketing Department G-Suite").
- `provider` (string, required): One of `google_workspace`, `microsoft_365`, or `generic_imap`.
- `providerConfig` (object, required): Configuration specific to the provider.
##### `providerConfig` for `google_workspace` / `microsoft_365`
```json
{
"name": "Corporate Google Workspace",
"provider": "google_workspace",
"providerConfig": {
"clientId": "your-oauth-client-id.apps.googleusercontent.com",
"clientSecret": "your-super-secret-client-secret",
"redirectUri": "https://yourapp.com/oauth/google/callback"
}
}
```
##### `providerConfig` for `generic_imap`
```json
{
"name": "Legacy IMAP Server",
"provider": "generic_imap",
"providerConfig": {
"host": "imap.example.com",
"port": 993,
"secure": true,
"username": "archive-user",
"password": "imap-password"
}
```typescript
interface CreateIngestionSourceDto {
name: string;
provider: 'google' | 'microsoft' | 'generic_imap';
providerConfig: IngestionCredentials;
}
```
#### Responses
- **Success (`201 Created`):** Returns the full `IngestionSource` object, which now includes a system-generated `id` and default status.
- **201 Created:** The newly created ingestion source.
- **500 Internal Server Error:** An unexpected error occurred.
```json
{
"id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
"name": "Corporate Google Workspace",
"provider": "google_workspace",
"status": "pending_auth",
"createdAt": "2025-07-11T12:00:00.000Z",
"updatedAt": "2025-07-11T12:00:00.000Z",
"providerConfig": { ... }
}
```
### GET /api/v1/ingestion
- **Error (`500 Internal Server Error`):** Indicates a server-side problem during creation.
Retrieves all ingestion sources.
---
## 2. Get All Ingestion Sources
- **Method:** `GET`
- **Path:** `/`
- **Description:** Retrieves a list of all configured ingestion sources for the organization.
**Access:** Authenticated
#### Responses
- **Success (`200 OK`):** Returns an array of `IngestionSource` objects.
- **200 OK:** An array of ingestion source objects.
- **500 Internal Server Error:** An unexpected error occurred.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.
### GET /api/v1/ingestion/:id
---
Retrieves a single ingestion source by its ID.
## 3. Get Ingestion Source by ID
- **Method:** `GET`
- **Path:** `/:id`
- **Description:** Fetches the details of a specific ingestion source.
**Access:** Authenticated
#### URL Parameters
- `id` (string, required): The UUID of the ingestion source.
| Parameter | Type | Description |
| :-------- | :----- | :------------------------------ |
| `id` | string | The ID of the ingestion source. |
#### Responses
- **Success (`200 OK`):** Returns the corresponding `IngestionSource` object.
- **200 OK:** The ingestion source object.
- **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred.
- **Error (`404 Not Found`):** Returned if no source with the given ID exists.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.
### PUT /api/v1/ingestion/:id
---
Updates an existing ingestion source.
## 4. Update Ingestion Source
- **Method:** `PUT`
- **Path:** `/:id`
- **Description:** Modifies an existing ingestion source. This is useful for changing the name, pausing a source, or updating its configuration.
**Access:** Authenticated
#### URL Parameters
- `id` (string, required): The UUID of the ingestion source to update.
| Parameter | Type | Description |
| :-------- | :----- | :------------------------------ |
| `id` | string | The ID of the ingestion source. |
#### Request Body (`UpdateIngestionSourceDto`)
#### Request Body
All fields are optional. Only include the fields you want to change.
The request body should be an `UpdateIngestionSourceDto` object.
```json
{
"name": "Marketing Dept G-Suite (Paused)",
"status": "paused"
```typescript
interface UpdateIngestionSourceDto {
name?: string;
provider?: 'google' | 'microsoft' | 'generic_imap';
providerConfig?: IngestionCredentials;
status?:
| 'pending_auth'
| 'auth_success'
| 'importing'
| 'active'
| 'paused'
| 'error';
}
```
#### Responses
- **Success (`200 OK`):** Returns the complete, updated `IngestionSource` object.
- **200 OK:** The updated ingestion source object.
- **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred.
- **Error (`404 Not Found`):** Returned if no source with the given ID exists.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.
### DELETE /api/v1/ingestion/:id
---
Deletes an ingestion source and all associated data.
## 5. Delete Ingestion Source
- **Method:** `DELETE`
- **Path:** `/:id`
- **Description:** Permanently removes an ingestion source. This action cannot be undone.
**Access:** Authenticated
#### URL Parameters
- `id` (string, required): The UUID of the ingestion source to delete.
| Parameter | Type | Description |
| :-------- | :----- | :------------------------------ |
| `id` | string | The ID of the ingestion source. |
#### Responses
- **Success (`204 No Content`):** Indicates successful deletion with no body content.
- **204 No Content:** The ingestion source was deleted successfully.
- **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred.
- **Error (`404 Not Found`):** Returned if no source with the given ID exists.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.
### POST /api/v1/ingestion/:id/import
---
Triggers the initial import process for an ingestion source.
## 6. Trigger Initial Import
- **Method:** `POST`
- **Path:** `/:id/sync`
- **Description:** Initiates the email import process for a given source. This is an asynchronous operation that enqueues a background job and immediately returns a response. The status of the source will be updated to `importing`.
**Access:** Authenticated
#### URL Parameters
- `id` (string, required): The UUID of the ingestion source to sync.
| Parameter | Type | Description |
| :-------- | :----- | :------------------------------ |
| `id` | string | The ID of the ingestion source. |
#### Responses
- **Success (`202 Accepted`):** Confirms that the sync request has been accepted for processing.
- **202 Accepted:** The initial import was triggered successfully.
- **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred.
```json
{
"message": "Initial import triggered successfully."
}
```
### POST /api/v1/ingestion/:id/pause
- **Error (`404 Not Found`):** Returned if no source with the given ID exists.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.
Pauses an active ingestion source.
**Access:** Authenticated
#### URL Parameters
| Parameter | Type | Description |
| :-------- | :----- | :------------------------------ |
| `id` | string | The ID of the ingestion source. |
#### Responses
- **200 OK:** The updated ingestion source object with a `paused` status.
- **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred.
### POST /api/v1/ingestion/:id/sync
Triggers a forced synchronization for an ingestion source.
**Access:** Authenticated
#### URL Parameters
| Parameter | Type | Description |
| :-------- | :----- | :------------------------------ |
| `id` | string | The ID of the ingestion source. |
#### Responses
- **202 Accepted:** The force sync was triggered successfully.
- **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred.

50
docs/api/search.md Normal file
View File

@@ -0,0 +1,50 @@
# Search Service API
The Search Service provides an endpoint for searching indexed emails.
## Endpoints
All endpoints in this service require authentication.
### GET /api/v1/search
Performs a search query against the indexed emails.
**Access:** Authenticated
#### Query Parameters
| Parameter | Type | Description | Default |
| :----------------- | :----- | :--------------------------------------------------------------------- | :------ |
| `keywords` | string | The search query. | |
| `page` | number | The page number for pagination. | 1 |
| `limit` | number | The number of items per page. | 10 |
| `matchingStrategy` | string | The matching strategy to use (`all` or `last`). | `last` |
| `filters` | object | Key-value pairs for filtering results (e.g., `from=user@example.com`). | |
#### Responses
- **200 OK:** A search result object.
```json
{
"hits": [
{
"id": "email-id",
"subject": "Test Email",
"from": "sender@example.com",
"_formatted": {
"subject": "<em>Test</em> Email"
}
}
],
"total": 1,
"page": 1,
"limit": 10,
"totalPages": 1,
"processingTimeMs": 5
}
```
- **400 Bad Request:** Keywords are required.
- **500 Internal Server Error:** An unexpected error occurred.

26
docs/api/storage.md Normal file
View File

@@ -0,0 +1,26 @@
# Storage Service API
The Storage Service provides an endpoint for downloading files from the configured storage provider.
## Endpoints
All endpoints in this service require authentication.
### GET /api/v1/storage/download
Downloads a file from the storage.
**Access:** Authenticated
#### Query Parameters
| Parameter | Type | Description |
| :-------- | :----- | :------------------------------------------------ |
| `path` | string | The path to the file within the storage provider. |
#### Responses
- **200 OK:** The file stream.
- **400 Bad Request:** File path is required or invalid.
- **404 Not Found:** File not found.
- **500 Internal Server Error:** An unexpected error occurred.

12
docs/index.md
View File

@@ -1,14 +1,14 @@
# Get Started
# Get Started 👋
Welcome to Open Archiver! This guide will help you get started with setting up and using the platform.
## What is Open Archiver?
## What is Open Archiver? 🛡️
**A secure, sovereign, and affordable open-source platform for email archiving and eDiscovery.**
Open Archiver provides a robust, self-hosted solution for archiving, storing, indexing, and searching emails from major platforms, including Google Workspace (Gmail), Microsoft 365, as well as generic IMAP-enabled email inboxes. Use Open Archiver to keep a permanent, tamper-proof record of your communication history, free from vendor lock-in.
## Key Features
## Key Features ✨
- **Universal Ingestion**: Connect to Google Workspace, Microsoft 365, and standard IMAP servers to perform initial bulk imports and maintain continuous, real-time synchronization.
- **Secure & Efficient Storage**: Emails are stored in the standard `.eml` format. The system uses deduplication and compression to minimize storage costs. All data is encrypted at rest.
@@ -17,13 +17,13 @@ Open Archiver provides a robust, self-hosted solution for archiving, storing, in
- **Compliance & Retention**: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD).
- **Comprehensive Auditing**: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when (TBD).
## Installation
## Installation 🚀
To get your own instance of Open Archiver running, follow our detailed installation guide:
- [Installation Guide](./user-guides/installation.md)
## Data Source Configuration
## Data Source Configuration 🔌
After deploying the application, you will need to configure one or more ingestion sources to begin archiving emails. Follow our detailed guides to connect to your email provider:
@@ -31,7 +31,7 @@ After deploying the application, you will need to configure one or more ingestio
- [Connecting to Microsoft 365](./user-guides/email-providers/microsoft-365.md)
- [Connecting to a Generic IMAP Server](./user-guides/email-providers/imap.md)
## Contributing
## Contributing ❤️
We welcome contributions from the community!