github/OpenArchiver
1
1
Fork 0
mirror of https://github.com/LogicLabs-OU/OpenArchiver.git synced 2026-04-06 00:31:57 +02:00
Code Issues 161 Packages Projects Releases Wiki Activity

Project wide format

Browse Source
This commit is contained in:
Wayne
2025-08-15 14:18:23 +03:00
parent 9873228d01
commit b2ca3ef0e1
163 changed files with 17955 additions and 18307 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
README.md
View File

@@ -40,38 +40,37 @@ Password: openarchiver_demo
## ✨ Key Features ## ✨ Key Features
- **Universal Ingestion**: Connect to any email provider to perform initial bulk imports and maintain continuous, real-time synchronization. Ingestion sources include: - **Universal Ingestion**: Connect to any email provider to perform initial bulk imports and maintain continuous, real-time synchronization. Ingestion sources include:
- IMAP connection
- Google Workspace
- Microsoft 365
- PST files
- Zipped .eml files
- IMAP connection - **Secure & Efficient Storage**: Emails are stored in the standard `.eml` format. The system uses deduplication and compression to minimize storage costs. All data is encrypted at rest.
- Google Workspace - **Pluggable Storage Backends**: Support both local filesystem storage and S3-compatible object storage (like AWS S3 or MinIO).
- Microsoft 365 - **Powerful Search & eDiscovery**: A high-performance search engine indexes the full text of emails and attachments (PDF, DOCX, etc.).
- PST files - **Thread discovery**: The ability to discover if an email belongs to a thread/conversation and present the context.
- Zipped .eml files - **Compliance & Retention**: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD).
- **Comprehensive Auditing**: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when (TBD).
- **Secure & Efficient Storage**: Emails are stored in the standard `.eml` format. The system uses deduplication and compression to minimize storage costs. All data is encrypted at rest.
- **Pluggable Storage Backends**: Support both local filesystem storage and S3-compatible object storage (like AWS S3 or MinIO).
- **Powerful Search & eDiscovery**: A high-performance search engine indexes the full text of emails and attachments (PDF, DOCX, etc.).
- **Thread discovery**: The ability to discover if an email belongs to a thread/conversation and present the context.
- **Compliance & Retention**: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD).
- **Comprehensive Auditing**: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when (TBD).
## 🛠️ Tech Stack ## 🛠️ Tech Stack
Open Archiver is built on a modern, scalable, and maintainable technology stack: Open Archiver is built on a modern, scalable, and maintainable technology stack:
- **Frontend**: SvelteKit with Svelte 5 - **Frontend**: SvelteKit with Svelte 5
- **Backend**: Node.js with Express.js & TypeScript - **Backend**: Node.js with Express.js & TypeScript
- **Job Queue**: BullMQ on Redis for robust, asynchronous processing. (We use Valkey as the Redis service in the Docker Compose deployment mode, but you can use Redis as well.) - **Job Queue**: BullMQ on Redis for robust, asynchronous processing. (We use Valkey as the Redis service in the Docker Compose deployment mode, but you can use Redis as well.)
- **Search Engine**: Meilisearch for blazingly fast and resource-efficient search - **Search Engine**: Meilisearch for blazingly fast and resource-efficient search
- **Database**: PostgreSQL for metadata, user management, and audit logs - **Database**: PostgreSQL for metadata, user management, and audit logs
- **Deployment**: Docker Compose deployment - **Deployment**: Docker Compose deployment
## 📦 Deployment ## 📦 Deployment
### Prerequisites ### Prerequisites
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) - [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/)
- A server or local machine with at least 4GB of RAM (2GB of RAM if you use external Postgres, Redis (Valkey) and Meilisearch instances). - A server or local machine with at least 4GB of RAM (2GB of RAM if you use external Postgres, Redis (Valkey) and Meilisearch instances).
### Installation ### Installation
@@ -106,17 +105,17 @@ Open Archiver is built on a modern, scalable, and maintainable technology stack:
After deploying the application, you will need to configure one or more ingestion sources to begin archiving emails. Follow our detailed guides to connect to your email provider: After deploying the application, you will need to configure one or more ingestion sources to begin archiving emails. Follow our detailed guides to connect to your email provider:
- [Connecting to Google Workspace](https://docs.openarchiver.com/user-guides/email-providers/google-workspace.html) - [Connecting to Google Workspace](https://docs.openarchiver.com/user-guides/email-providers/google-workspace.html)
- [Connecting to Microsoft 365](https://docs.openarchiver.com/user-guides/email-providers/imap.html) - [Connecting to Microsoft 365](https://docs.openarchiver.com/user-guides/email-providers/imap.html)
- [Connecting to a Generic IMAP Server](https://docs.openarchiver.com/user-guides/email-providers/imap.html) - [Connecting to a Generic IMAP Server](https://docs.openarchiver.com/user-guides/email-providers/imap.html)
## 🤝 Contributing ## 🤝 Contributing
We welcome contributions from the community! We welcome contributions from the community!
- **Reporting Bugs**: If you find a bug, please open an issue on our GitHub repository. - **Reporting Bugs**: If you find a bug, please open an issue on our GitHub repository.
- **Suggesting Enhancements**: Have an idea for a new feature? We'd love to hear it. Open an issue to start the discussion. - **Suggesting Enhancements**: Have an idea for a new feature? We'd love to hear it. Open an issue to start the discussion.
- **Code Contributions**: If you'd like to contribute code, please fork the repository and submit a pull request. - **Code Contributions**: If you'd like to contribute code, please fork the repository and submit a pull request.
Please read our `CONTRIBUTING.md` file for more details on our code of conduct and the process for submitting pull requests. Please read our `CONTRIBUTING.md` file for more details on our code of conduct and the process for submitting pull requests.

128
docker-compose.yml
View File

@@ -1,74 +1,74 @@
version: '3.8' version: '3.8'
services: services:
open-archiver: open-archiver:
image: logiclabshq/open-archiver:latest image: logiclabshq/open-archiver:latest
container_name: open-archiver container_name: open-archiver
restart: unless-stopped restart: unless-stopped
ports: ports:
- '4000:4000' # Backend - '4000:4000' # Backend
- '3000:3000' # Frontend - '3000:3000' # Frontend
env_file: env_file:
- .env - .env
volumes: volumes:
- archiver-data:/var/data/open-archiver - archiver-data:/var/data/open-archiver
depends_on: depends_on:
- postgres - postgres
- valkey - valkey
- meilisearch - meilisearch
networks: networks:
- open-archiver-net - open-archiver-net
postgres: postgres:
image: postgres:17-alpine image: postgres:17-alpine
container_name: postgres container_name: postgres
restart: unless-stopped restart: unless-stopped
environment: environment:
POSTGRES_DB: ${POSTGRES_DB:-open_archive} POSTGRES_DB: ${POSTGRES_DB:-open_archive}
POSTGRES_USER: ${POSTGRES_USER:-admin} POSTGRES_USER: ${POSTGRES_USER:-admin}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-password} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-password}
volumes: volumes:
- pgdata:/var/lib/postgresql/data - pgdata:/var/lib/postgresql/data
ports: ports:
- '5432:5432' - '5432:5432'
networks: networks:
- open-archiver-net - open-archiver-net
valkey: valkey:
image: valkey/valkey:8-alpine image: valkey/valkey:8-alpine
container_name: valkey container_name: valkey
restart: unless-stopped restart: unless-stopped
command: valkey-server --requirepass ${REDIS_PASSWORD} command: valkey-server --requirepass ${REDIS_PASSWORD}
ports: ports:
- '6379:6379' - '6379:6379'
volumes: volumes:
- valkeydata:/data - valkeydata:/data
networks: networks:
- open-archiver-net - open-archiver-net
meilisearch: meilisearch:
image: getmeili/meilisearch:v1.15 image: getmeili/meilisearch:v1.15
container_name: meilisearch container_name: meilisearch
restart: unless-stopped restart: unless-stopped
environment: environment:
MEILI_MASTER_KEY: ${MEILI_MASTER_KEY:-aSampleMasterKey} MEILI_MASTER_KEY: ${MEILI_MASTER_KEY:-aSampleMasterKey}
ports: ports:
- '7700:7700' - '7700:7700'
volumes: volumes:
- meilidata:/meili_data - meilidata:/meili_data
networks: networks:
- open-archiver-net - open-archiver-net
volumes: volumes:
pgdata: pgdata:
driver: local driver: local
valkeydata: valkeydata:
driver: local driver: local
meilidata: meilidata:
driver: local driver: local
archiver-data: archiver-data:
driver: local driver: local
networks: networks:
open-archiver-net: open-archiver-net:
driver: bridge driver: bridge

143
docs/.vitepress/config.mts
View File

@@ -1,71 +1,80 @@
import { defineConfig } from 'vitepress'; import { defineConfig } from 'vitepress';
export default defineConfig({ export default defineConfig({
head: [ head: [
[ [
'script', 'script',
{ {
defer: '', defer: '',
src: 'https://analytics.zenceipt.com/script.js', src: 'https://analytics.zenceipt.com/script.js',
'data-website-id': '2c8b452e-eab5-4f82-8ead-902d8f8b976f' 'data-website-id': '2c8b452e-eab5-4f82-8ead-902d8f8b976f',
} },
] ],
], ],
title: 'Open Archiver', title: 'Open Archiver',
description: 'Official documentation for the Open Archiver project.', description: 'Official documentation for the Open Archiver project.',
themeConfig: { themeConfig: {
search: { search: {
provider: 'local' provider: 'local',
}, },
logo: { logo: {
src: '/logo-sq.svg' src: '/logo-sq.svg',
}, },
nav: [ nav: [
{ text: 'Home', link: '/' }, { text: 'Home', link: '/' },
{ text: 'Github', link: 'https://github.com/LogicLabs-OU/OpenArchiver' }, { text: 'Github', link: 'https://github.com/LogicLabs-OU/OpenArchiver' },
{ text: "Website", link: 'https://openarchiver.com/' }, { text: 'Website', link: 'https://openarchiver.com/' },
{ text: "Discord", link: 'https://discord.gg/MTtD7BhuTQ' } { text: 'Discord', link: 'https://discord.gg/MTtD7BhuTQ' },
], ],
sidebar: [ sidebar: [
{ {
text: 'User Guides', text: 'User Guides',
items: [ items: [
{ text: 'Get Started', link: '/' }, { text: 'Get Started', link: '/' },
{ text: 'Installation', link: '/user-guides/installation' }, { text: 'Installation', link: '/user-guides/installation' },
{ {
text: 'Email Providers', text: 'Email Providers',
link: '/user-guides/email-providers/', link: '/user-guides/email-providers/',
collapsed: true, collapsed: true,
items: [ items: [
{ text: 'Generic IMAP Server', link: '/user-guides/email-providers/imap' }, {
{ text: 'Google Workspace', link: '/user-guides/email-providers/google-workspace' }, text: 'Generic IMAP Server',
{ text: 'Microsoft 365', link: '/user-guides/email-providers/microsoft-365' }, link: '/user-guides/email-providers/imap',
{ text: 'EML Import', link: '/user-guides/email-providers/eml' }, },
{ text: 'PST Import', link: '/user-guides/email-providers/pst' } {
] text: 'Google Workspace',
} link: '/user-guides/email-providers/google-workspace',
] },
}, {
{ text: 'Microsoft 365',
text: 'API Reference', link: '/user-guides/email-providers/microsoft-365',
items: [ },
{ text: 'Overview', link: '/api/' }, { text: 'EML Import', link: '/user-guides/email-providers/eml' },
{ text: 'Authentication', link: '/api/authentication' }, { text: 'PST Import', link: '/user-guides/email-providers/pst' },
{ text: 'Auth', link: '/api/auth' }, ],
{ text: 'Archived Email', link: '/api/archived-email' }, },
{ text: 'Dashboard', link: '/api/dashboard' }, ],
{ text: 'Ingestion', link: '/api/ingestion' }, },
{ text: 'Search', link: '/api/search' }, {
{ text: 'Storage', link: '/api/storage' } text: 'API Reference',
] items: [
}, { text: 'Overview', link: '/api/' },
{ { text: 'Authentication', link: '/api/authentication' },
text: 'Services', { text: 'Auth', link: '/api/auth' },
items: [ { text: 'Archived Email', link: '/api/archived-email' },
{ text: 'Overview', link: '/services/' }, { text: 'Dashboard', link: '/api/dashboard' },
{ text: 'Storage Service', link: '/services/storage-service' } { text: 'Ingestion', link: '/api/ingestion' },
] { text: 'Search', link: '/api/search' },
} { text: 'Storage', link: '/api/storage' },
] ],
} },
{
text: 'Services',
items: [
{ text: 'Overview', link: '/services/' },
{ text: 'Storage Service', link: '/services/storage-service' },
],
},
],
},
}); });

20
docs/SUMMARY.md
View File

@@ -2,16 +2,16 @@
## User guides ## User guides
- [Get started](index.md) - [Get started](index.md)
- [Installation](user-guides/installation.md) - [Installation](user-guides/installation.md)
- [email-providers](user-guides/email-providers/index.md) - [email-providers](user-guides/email-providers/index.md)
- [Connecting to Google Workspace](user-guides/email-providers/google-workspace.md) - [Connecting to Google Workspace](user-guides/email-providers/google-workspace.md)
- [Connecting to a Generic IMAP Server](user-guides/email-providers/imap.md) - [Connecting to a Generic IMAP Server](user-guides/email-providers/imap.md)
- [Connecting to Microsoft 365](user-guides/email-providers/microsoft-365.md) - [Connecting to Microsoft 365](user-guides/email-providers/microsoft-365.md)
--- ---
- [api](api/index.md) - [api](api/index.md)
- [Ingestion Sources API Documentation](api/ingestion.md) - [Ingestion Sources API Documentation](api/ingestion.md)
- [services](services/index.md) - [services](services/index.md)
- [Pluggable Storage Service (StorageService)](services/storage-service.md) - [Pluggable Storage Service (StorageService)](services/storage-service.md)

82
docs/api/archived-email.md
View File

@@ -27,29 +27,27 @@ Retrieves a paginated list of archived emails for a specific ingestion source.
#### Responses #### Responses
- **200 OK:** A paginated list of archived emails. - **200 OK:** A paginated list of archived emails.
```json ```json
{ {
"items": [ "items": [
{ {
"id": "email-id", "id": "email-id",
"subject": "Test Email", "subject": "Test Email",
"from": "sender@example.com", "from": "sender@example.com",
"sentAt": "2023-10-27T10:00:00.000Z", "sentAt": "2023-10-27T10:00:00.000Z",
"hasAttachments": true, "hasAttachments": true,
"recipients": [ "recipients": [{ "name": "Recipient 1", "email": "recipient1@example.com" }]
{ "name": "Recipient 1", "email": "recipient1@example.com" } }
] ],
} "total": 100,
], "page": 1,
"total": 100, "limit": 10
"page": 1,
"limit": 10
} }
``` ```
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
### GET /api/v1/archived-emails/:id ### GET /api/v1/archived-emails/:id
@@ -65,32 +63,30 @@ Retrieves a single archived email by its ID, including its raw content and attac
#### Responses #### Responses
- **200 OK:** The archived email details. - **200 OK:** The archived email details.
```json ```json
{ {
"id": "email-id", "id": "email-id",
"subject": "Test Email", "subject": "Test Email",
"from": "sender@example.com", "from": "sender@example.com",
"sentAt": "2023-10-27T10:00:00.000Z", "sentAt": "2023-10-27T10:00:00.000Z",
"hasAttachments": true, "hasAttachments": true,
"recipients": [ "recipients": [{ "name": "Recipient 1", "email": "recipient1@example.com" }],
{ "name": "Recipient 1", "email": "recipient1@example.com" } "raw": "...",
], "attachments": [
"raw": "...", {
"attachments": [ "id": "attachment-id",
{ "filename": "document.pdf",
"id": "attachment-id", "mimeType": "application/pdf",
"filename": "document.pdf", "sizeBytes": 12345
"mimeType": "application/pdf", }
"sizeBytes": 12345 ]
}
]
} }
``` ```
- **404 Not Found:** The archived email with the specified ID was not found. - **404 Not Found:** The archived email with the specified ID was not found.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
## Service Methods ## Service Methods
@@ -98,14 +94,14 @@ Retrieves a single archived email by its ID, including its raw content and attac
Retrieves a paginated list of archived emails from the database for a given ingestion source. Retrieves a paginated list of archived emails from the database for a given ingestion source.
- **ingestionSourceId:** The ID of the ingestion source. - **ingestionSourceId:** The ID of the ingestion source.
- **page:** The page number for pagination. - **page:** The page number for pagination.
- **limit:** The number of items per page. - **limit:** The number of items per page.
- **Returns:** A promise that resolves to a `PaginatedArchivedEmails` object. - **Returns:** A promise that resolves to a `PaginatedArchivedEmails` object.
### `getArchivedEmailById(emailId: string): Promise<ArchivedEmail | null>` ### `getArchivedEmailById(emailId: string): Promise<ArchivedEmail | null>`
Retrieves a single archived email by its ID, including its raw content and attachments. Retrieves a single archived email by its ID, including its raw content and attachments.
- **emailId:** The ID of the archived email. - **emailId:** The ID of the archived email.
- **Returns:** A promise that resolves to an `ArchivedEmail` object or `null` if not found. - **Returns:** A promise that resolves to an `ArchivedEmail` object or `null` if not found.

42
docs/api/auth.md
View File

@@ -21,40 +21,40 @@ Authenticates a user and returns a JWT if the credentials are valid.
#### Responses #### Responses
- **200 OK:** Authentication successful. - **200 OK:** Authentication successful.
```json ```json
{ {
"accessToken": "your.jwt.token", "accessToken": "your.jwt.token",
"user": { "user": {
"id": "user-id", "id": "user-id",
"email": "user@example.com", "email": "user@example.com",
"role": "user" "role": "user"
} }
} }
``` ```
- **400 Bad Request:** Email or password not provided. - **400 Bad Request:** Email or password not provided.
```json ```json
{ {
"message": "Email and password are required" "message": "Email and password are required"
} }
``` ```
- **401 Unauthorized:** Invalid credentials. - **401 Unauthorized:** Invalid credentials.
```json ```json
{ {
"message": "Invalid credentials" "message": "Invalid credentials"
} }
``` ```
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
```json ```json
{ {
"message": "An internal server error occurred" "message": "An internal server error occurred"
} }
``` ```
@@ -64,21 +64,21 @@ Authenticates a user and returns a JWT if the credentials are valid.
Compares a plain-text password with a hashed password to verify its correctness. Compares a plain-text password with a hashed password to verify its correctness.
- **password:** The plain-text password. - **password:** The plain-text password.
- **hash:** The hashed password to compare against. - **hash:** The hashed password to compare against.
- **Returns:** A promise that resolves to `true` if the password is valid, otherwise `false`. - **Returns:** A promise that resolves to `true` if the password is valid, otherwise `false`.
### `login(email: string, password: string): Promise<LoginResponse | null>` ### `login(email: string, password: string): Promise<LoginResponse | null>`
Handles the user login process. It finds the user by email, verifies the password, and generates a JWT upon successful authentication. Handles the user login process. It finds the user by email, verifies the password, and generates a JWT upon successful authentication.
- **email:** The user's email. - **email:** The user's email.
- **password:** The user's password. - **password:** The user's password.
- **Returns:** A promise that resolves to a `LoginResponse` object containing the `accessToken` and `user` details, or `null` if authentication fails. - **Returns:** A promise that resolves to a `LoginResponse` object containing the `accessToken` and `user` details, or `null` if authentication fails.
### `verifyToken(token: string): Promise<AuthTokenPayload | null>` ### `verifyToken(token: string): Promise<AuthTokenPayload | null>`
Verifies the authenticity and expiration of a JWT. Verifies the authenticity and expiration of a JWT.
- **token:** The JWT string to verify. - **token:** The JWT string to verify.
- **Returns:** A promise that resolves to the token's `AuthTokenPayload` if valid, otherwise `null`. - **Returns:** A promise that resolves to the token's `AuthTokenPayload` if valid, otherwise `null`.

12
docs/api/authentication.md
View File

@@ -22,12 +22,12 @@ Content-Type: application/json
```json ```json
{ {
"accessToken": "your.jwt.token", "accessToken": "your.jwt.token",
"user": { "user": {
"id": "user-id", "id": "user-id",
"email": "user@example.com", "email": "user@example.com",
"role": "user" "role": "user"
} }
} }
``` ```

76
docs/api/dashboard.md
View File

@@ -14,13 +14,13 @@ Retrieves overall statistics, including the total number of archived emails, tot
#### Responses #### Responses
- **200 OK:** An object containing the dashboard statistics. - **200 OK:** An object containing the dashboard statistics.
```json ```json
{ {
"totalEmailsArchived": 12345, "totalEmailsArchived": 12345,
"totalStorageUsed": 54321098, "totalStorageUsed": 54321098,
"failedIngestionsLast7Days": 3 "failedIngestionsLast7Days": 3
} }
``` ```
@@ -32,20 +32,20 @@ Retrieves the email ingestion history for the last 30 days, grouped by day.
#### Responses #### Responses
- **200 OK:** An object containing the ingestion history. - **200 OK:** An object containing the ingestion history.
```json ```json
{ {
"history": [ "history": [
{ {
"date": "2023-09-27T00:00:00.000Z", "date": "2023-09-27T00:00:00.000Z",
"count": 150 "count": 150
}, },
{ {
"date": "2023-09-28T00:00:00.000Z", "date": "2023-09-28T00:00:00.000Z",
"count": 200 "count": 200
} }
] ]
} }
``` ```
@@ -57,24 +57,24 @@ Retrieves a list of all ingestion sources along with their status and storage us
#### Responses #### Responses
- **200 OK:** An array of ingestion source objects. - **200 OK:** An array of ingestion source objects.
```json ```json
[ [
{ {
"id": "source-id-1", "id": "source-id-1",
"name": "Google Workspace", "name": "Google Workspace",
"provider": "google", "provider": "google",
"status": "active", "status": "active",
"storageUsed": 12345678 "storageUsed": 12345678
}, },
{ {
"id": "source-id-2", "id": "source-id-2",
"name": "Microsoft 365", "name": "Microsoft 365",
"provider": "microsoft", "provider": "microsoft",
"status": "error", "status": "error",
"storageUsed": 87654321 "storageUsed": 87654321
} }
] ]
``` ```
@@ -86,7 +86,7 @@ Retrieves a list of recent synchronization jobs. (Note: This is currently a plac
#### Responses #### Responses
- **200 OK:** An empty array. - **200 OK:** An empty array.
```json ```json
[] []
@@ -100,15 +100,15 @@ Retrieves insights from the indexed email data, such as the top senders.
#### Responses #### Responses
- **200 OK:** An object containing indexed insights. - **200 OK:** An object containing indexed insights.
```json ```json
{ {
"topSenders": [ "topSenders": [
{ {
"sender": "user@example.com", "sender": "user@example.com",
"count": 42 "count": 42
} }
] ]
} }
``` ```

12
docs/api/index.md
View File

@@ -10,9 +10,9 @@ Before making requests to protected endpoints, you must authenticate with the AP
## API Services ## API Services
- [**Auth Service**](./auth.md): Handles user authentication. - [**Auth Service**](./auth.md): Handles user authentication.
- [**Archived Email Service**](./archived-email.md): Manages archived emails. - [**Archived Email Service**](./archived-email.md): Manages archived emails.
- [**Dashboard Service**](./dashboard.md): Provides data for the main dashboard. - [**Dashboard Service**](./dashboard.md): Provides data for the main dashboard.
- [**Ingestion Service**](./ingestion.md): Manages email ingestion sources. - [**Ingestion Service**](./ingestion.md): Manages email ingestion sources.
- [**Search Service**](./search.md): Handles email search functionality. - [**Search Service**](./search.md): Handles email search functionality.
- [**Storage Service**](./storage.md): Manages file storage and downloads. - [**Storage Service**](./storage.md): Manages file storage and downloads.

64
docs/api/ingestion.md
View File

@@ -18,16 +18,16 @@ The request body should be a `CreateIngestionSourceDto` object.
```typescript ```typescript
interface CreateIngestionSourceDto { interface CreateIngestionSourceDto {
name: string; name: string;
provider: 'google' | 'microsoft' | 'generic_imap'; provider: 'google' | 'microsoft' | 'generic_imap';
providerConfig: IngestionCredentials; providerConfig: IngestionCredentials;
} }
``` ```
#### Responses #### Responses
- **201 Created:** The newly created ingestion source. - **201 Created:** The newly created ingestion source.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
### GET /api/v1/ingestion-sources ### GET /api/v1/ingestion-sources
@@ -37,8 +37,8 @@ Retrieves all ingestion sources.
#### Responses #### Responses
- **200 OK:** An array of ingestion source objects. - **200 OK:** An array of ingestion source objects.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
### GET /api/v1/ingestion-sources/:id ### GET /api/v1/ingestion-sources/:id
@@ -54,9 +54,9 @@ Retrieves a single ingestion source by its ID.
#### Responses #### Responses
- **200 OK:** The ingestion source object. - **200 OK:** The ingestion source object.
- **404 Not Found:** Ingestion source not found. - **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
### PUT /api/v1/ingestion-sources/:id ### PUT /api/v1/ingestion-sources/:id
@@ -76,24 +76,18 @@ The request body should be an `UpdateIngestionSourceDto` object.
```typescript ```typescript
interface UpdateIngestionSourceDto { interface UpdateIngestionSourceDto {
name?: string; name?: string;
provider?: 'google' | 'microsoft' | 'generic_imap'; provider?: 'google' | 'microsoft' | 'generic_imap';
providerConfig?: IngestionCredentials; providerConfig?: IngestionCredentials;
status?: status?: 'pending_auth' | 'auth_success' | 'importing' | 'active' | 'paused' | 'error';
| 'pending_auth'
| 'auth_success'
| 'importing'
| 'active'
| 'paused'
| 'error';
} }
``` ```
#### Responses #### Responses
- **200 OK:** The updated ingestion source object. - **200 OK:** The updated ingestion source object.
- **404 Not Found:** Ingestion source not found. - **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
### DELETE /api/v1/ingestion-sources/:id ### DELETE /api/v1/ingestion-sources/:id
@@ -109,9 +103,9 @@ Deletes an ingestion source and all associated data.
#### Responses #### Responses
- **204 No Content:** The ingestion source was deleted successfully. - **204 No Content:** The ingestion source was deleted successfully.
- **404 Not Found:** Ingestion source not found. - **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
### POST /api/v1/ingestion-sources/:id/import ### POST /api/v1/ingestion-sources/:id/import
@@ -127,9 +121,9 @@ Triggers the initial import process for an ingestion source.
#### Responses #### Responses
- **202 Accepted:** The initial import was triggered successfully. - **202 Accepted:** The initial import was triggered successfully.
- **404 Not Found:** Ingestion source not found. - **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
### POST /api/v1/ingestion-sources/:id/pause ### POST /api/v1/ingestion-sources/:id/pause
@@ -145,9 +139,9 @@ Pauses an active ingestion source.
#### Responses #### Responses
- **200 OK:** The updated ingestion source object with a `paused` status. - **200 OK:** The updated ingestion source object with a `paused` status.
- **404 Not Found:** Ingestion source not found. - **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.
### POST /api/v1/ingestion-sources/:id/sync ### POST /api/v1/ingestion-sources/:id/sync
@@ -163,6 +157,6 @@ Triggers a forced synchronization for an ingestion source.
#### Responses #### Responses
- **202 Accepted:** The force sync was triggered successfully. - **202 Accepted:** The force sync was triggered successfully.
- **404 Not Found:** Ingestion source not found. - **404 Not Found:** Ingestion source not found.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.

36
docs/api/search.md
View File

@@ -24,27 +24,27 @@ Performs a search query against the indexed emails.
#### Responses #### Responses
- **200 OK:** A search result object. - **200 OK:** A search result object.
```json ```json
{ {
"hits": [ "hits": [
{ {
"id": "email-id", "id": "email-id",
"subject": "Test Email", "subject": "Test Email",
"from": "sender@example.com", "from": "sender@example.com",
"_formatted": { "_formatted": {
"subject": "<em>Test</em> Email" "subject": "<em>Test</em> Email"
} }
} }
], ],
"total": 1, "total": 1,
"page": 1, "page": 1,
"limit": 10, "limit": 10,
"totalPages": 1, "totalPages": 1,
"processingTimeMs": 5 "processingTimeMs": 5
} }
``` ```
- **400 Bad Request:** Keywords are required. - **400 Bad Request:** Keywords are required.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.

8
docs/api/storage.md
View File

@@ -20,7 +20,7 @@ Downloads a file from the storage.
#### Responses #### Responses
- **200 OK:** The file stream. - **200 OK:** The file stream.
- **400 Bad Request:** File path is required or invalid. - **400 Bad Request:** File path is required or invalid.
- **404 Not Found:** File not found. - **404 Not Found:** File not found.
- **500 Internal Server Error:** An unexpected error occurred. - **500 Internal Server Error:** An unexpected error occurred.

26
docs/index.md
View File

@@ -10,33 +10,33 @@ Open Archiver provides a robust, self-hosted solution for archiving, storing, in
## Key Features ✨ ## Key Features ✨
- **Universal Ingestion**: Connect to Google Workspace, Microsoft 365, and standard IMAP servers to perform initial bulk imports and maintain continuous, real-time synchronization. - **Universal Ingestion**: Connect to Google Workspace, Microsoft 365, and standard IMAP servers to perform initial bulk imports and maintain continuous, real-time synchronization.
- **Secure & Efficient Storage**: Emails are stored in the standard `.eml` format. The system uses deduplication and compression to minimize storage costs. All data is encrypted at rest. - **Secure & Efficient Storage**: Emails are stored in the standard `.eml` format. The system uses deduplication and compression to minimize storage costs. All data is encrypted at rest.
- **Pluggable Storage Backends**: Support both local filesystem storage and S3-compatible object storage (like AWS S3 or MinIO). - **Pluggable Storage Backends**: Support both local filesystem storage and S3-compatible object storage (like AWS S3 or MinIO).
- **Powerful Search & eDiscovery**: A high-performance search engine indexes the full text of emails and attachments (PDF, DOCX, etc.). - **Powerful Search & eDiscovery**: A high-performance search engine indexes the full text of emails and attachments (PDF, DOCX, etc.).
- **Compliance & Retention**: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD). - **Compliance & Retention**: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD).
- **Comprehensive Auditing**: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when (TBD). - **Comprehensive Auditing**: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when (TBD).
## Installation 🚀 ## Installation 🚀
To get your own instance of Open Archiver running, follow our detailed installation guide: To get your own instance of Open Archiver running, follow our detailed installation guide:
- [Installation Guide](./user-guides/installation.md) - [Installation Guide](./user-guides/installation.md)
## Data Source Configuration 🔌 ## Data Source Configuration 🔌
After deploying the application, you will need to configure one or more ingestion sources to begin archiving emails. Follow our detailed guides to connect to your email provider: After deploying the application, you will need to configure one or more ingestion sources to begin archiving emails. Follow our detailed guides to connect to your email provider:
- [Connecting to Google Workspace](./user-guides/email-providers/google-workspace.md) - [Connecting to Google Workspace](./user-guides/email-providers/google-workspace.md)
- [Connecting to Microsoft 365](./user-guides/email-providers/microsoft-365.md) - [Connecting to Microsoft 365](./user-guides/email-providers/microsoft-365.md)
- [Connecting to a Generic IMAP Server](./user-guides/email-providers/imap.md) - [Connecting to a Generic IMAP Server](./user-guides/email-providers/imap.md)
## Contributing ❤️ ## Contributing ❤️
We welcome contributions from the community! We welcome contributions from the community!
- **Reporting Bugs**: If you find a bug, please open an issue on our GitHub repository. - **Reporting Bugs**: If you find a bug, please open an issue on our GitHub repository.
- **Suggesting Enhancements**: Have an idea for a new feature? We'd love to hear it. Open an issue to start the discussion. - **Suggesting Enhancements**: Have an idea for a new feature? We'd love to hear it. Open an issue to start the discussion.
- **Code Contributions**: If you'd like to contribute code, please fork the repository and submit a pull request. - **Code Contributions**: If you'd like to contribute code, please fork the repository and submit a pull request.
Please read our `CONTRIBUTING.md` file for more details on our code of conduct and the process for submitting pull requests. Please read our `CONTRIBUTING.md` file for more details on our code of conduct and the process for submitting pull requests.

20
docs/services/IAM-service/iam-policies.md
View File

@@ -8,15 +8,15 @@ A policy is a JSON object that consists of one or more statements. Each statemen
```json ```json
{ {
"Effect": "Allow", "Effect": "Allow",
"Action": ["archive:read", "archive:search"], "Action": ["archive:read", "archive:search"],
"Resource": ["archive/all"] "Resource": ["archive/all"]
} }
``` ```
- **`Effect`**: Specifies whether the statement results in an `Allow` or `Deny`. An explicit `Deny` always overrides an `Allow`. - **`Effect`**: Specifies whether the statement results in an `Allow` or `Deny`. An explicit `Deny` always overrides an `Allow`.
- **`Action`**: A list of operations that the policy grants or denies permission to perform. Actions are formatted as `service:operation`. - **`Action`**: A list of operations that the policy grants or denies permission to perform. Actions are formatted as `service:operation`.
- **`Resource`**: A list of resources to which the actions apply. Resources are specified in a hierarchical format. Wildcards (`*`) can be used. - **`Resource`**: A list of resources to which the actions apply. Resources are specified in a hierarchical format. Wildcards (`*`) can be used.
## 2. Wildcard Support ## 2. Wildcard Support
@@ -26,11 +26,11 @@ Our IAM system supports wildcards (`*`) in both `Action` and `Resource` fields t
You can use wildcards to grant broad permissions for actions: You can use wildcards to grant broad permissions for actions:
- **Global Wildcard (`*`)**: A standalone `*` in the `Action` field grants permission for all possible actions across all services. - **Global Wildcard (`*`)**: A standalone `*` in the `Action` field grants permission for all possible actions across all services.
```json ```json
"Action": ["*"] "Action": ["*"]
``` ```
- **Service-Level Wildcard (`service:*`)**: A wildcard at the end of an action string grants permission for all actions within that specific service. - **Service-Level Wildcard (`service:*`)**: A wildcard at the end of an action string grants permission for all actions within that specific service.
```json ```json
"Action": ["archive:*"] "Action": ["archive:*"]
``` ```
@@ -39,11 +39,11 @@ You can use wildcards to grant broad permissions for actions:
Wildcards can also be used to specify resources: Wildcards can also be used to specify resources:
- **Global Wildcard (`*`)**: A standalone `*` in the `Resource` field applies the policy to all resources in the system. - **Global Wildcard (`*`)**: A standalone `*` in the `Resource` field applies the policy to all resources in the system.
```json ```json
"Resource": ["*"] "Resource": ["*"]
``` ```
- **Partial Wildcards**: Some services allow wildcards at specific points in the resource path to refer to all resources of a certain type. For example, to target all ingestion sources: - **Partial Wildcards**: Some services allow wildcards at specific points in the resource path to refer to all resources of a certain type. For example, to target all ingestion sources:
```json ```json
"Resource": ["ingestion-source/*"] "Resource": ["ingestion-source/*"]
``` ```

1
docs/services/index.md
View File

@@ -1,2 +1 @@
# services # services

78
docs/services/storage-service.md
View File

@@ -14,8 +14,8 @@ The `StorageService` is configured via environment variables in the `.env` file.
The `STORAGE_TYPE` variable determines which provider the service will use. The `STORAGE_TYPE` variable determines which provider the service will use.
- `STORAGE_TYPE=local`: Uses the local server's filesystem. - `STORAGE_TYPE=local`: Uses the local server's filesystem.
- `STORAGE_TYPE=s3`: Uses an S3-compatible object storage service (e.g., AWS S3, MinIO, Google Cloud Storage). - `STORAGE_TYPE=s3`: Uses an S3-compatible object storage service (e.g., AWS S3, MinIO, Google Cloud Storage).
### 2. Local Filesystem Configuration ### 2. Local Filesystem Configuration
@@ -27,7 +27,7 @@ STORAGE_TYPE=local
STORAGE_LOCAL_ROOT_PATH=/var/data/open-archiver STORAGE_LOCAL_ROOT_PATH=/var/data/open-archiver
``` ```
- `STORAGE_LOCAL_ROOT_PATH`: The absolute path on the server where the archive will be created. The service will create subdirectories within this path as needed. - `STORAGE_LOCAL_ROOT_PATH`: The absolute path on the server where the archive will be created. The service will create subdirectories within this path as needed.
### 3. S3-Compatible Storage Configuration ### 3. S3-Compatible Storage Configuration
@@ -44,12 +44,12 @@ STORAGE_S3_REGION=us-east-1
STORAGE_S3_FORCE_PATH_STYLE=true STORAGE_S3_FORCE_PATH_STYLE=true
``` ```
- `STORAGE_S3_ENDPOINT`: The full URL of the S3 API endpoint. - `STORAGE_S3_ENDPOINT`: The full URL of the S3 API endpoint.
- `STORAGE_S3_BUCKET`: The name of the bucket to use for storage. - `STORAGE_S3_BUCKET`: The name of the bucket to use for storage.
- `STORAGE_S3_ACCESS_KEY_ID`: The access key for your S3 user. - `STORAGE_S3_ACCESS_KEY_ID`: The access key for your S3 user.
- `STORAGE_S3_SECRET_ACCESS_KEY`: The secret key for your S3 user. - `STORAGE_S3_SECRET_ACCESS_KEY`: The secret key for your S3 user.
- `STORAGE_S3_REGION` (Optional): The AWS region of your bucket. Recommended for AWS S3. - `STORAGE_S3_REGION` (Optional): The AWS region of your bucket. Recommended for AWS S3.
- `STORAGE_S3_FORCE_PATH_STYLE` (Optional): Set to `true` when using non-AWS S3 services like MinIO. - `STORAGE_S3_FORCE_PATH_STYLE` (Optional): Set to `true` when using non-AWS S3 services like MinIO.
## How to Use the Service ## How to Use the Service
@@ -61,31 +61,27 @@ The `StorageService` is designed to be used via dependency injection in other se
import { StorageService } from './StorageService'; import { StorageService } from './StorageService';
class IngestionService { class IngestionService {
private storageService: StorageService; private storageService: StorageService;
constructor() { constructor() {
// The StorageService is instantiated without any arguments. // The StorageService is instantiated without any arguments.
// It automatically reads the configuration from the environment. // It automatically reads the configuration from the environment.
this.storageService = new StorageService(); this.storageService = new StorageService();
} }
public async archiveEmail( public async archiveEmail(rawEmail: Buffer, userId: string, messageId: string): Promise<void> {
rawEmail: Buffer, // Define a structured, unique path for the email.
userId: string, const archivePath = `${userId}/messages/${messageId}.eml`;
messageId: string
): Promise<void> {
// Define a structured, unique path for the email.
const archivePath = `${userId}/messages/${messageId}.eml`;
try { try {
// Use the service. It doesn't know or care if this is writing // Use the service. It doesn't know or care if this is writing
// to a local disk or an S3 bucket. // to a local disk or an S3 bucket.
await this.storageService.put(archivePath, rawEmail); await this.storageService.put(archivePath, rawEmail);
console.log(`Successfully archived email to ${archivePath}`); console.log(`Successfully archived email to ${archivePath}`);
} catch (error) { } catch (error) {
console.error(`Failed to archive email ${messageId}`, error); console.error(`Failed to archive email ${messageId}`, error);
} }
} }
} }
``` ```
@@ -99,9 +95,9 @@ The `StorageService` implements the `IStorageProvider` interface. All methods ar
Stores a file at the specified path. If a file already exists at that path, it will be overwritten. Stores a file at the specified path. If a file already exists at that path, it will be overwritten.
- **`path: string`**: A unique identifier for the file, including its directory structure (e.g., `"user-123/emails/message-abc.eml"`). - **`path: string`**: A unique identifier for the file, including its directory structure (e.g., `"user-123/emails/message-abc.eml"`).
- **`content: Buffer | NodeJS.ReadableStream`**: The content of the file. It can be a `Buffer` for small files or a `ReadableStream` for large files to ensure memory efficiency. - **`content: Buffer | NodeJS.ReadableStream`**: The content of the file. It can be a `Buffer` for small files or a `ReadableStream` for large files to ensure memory efficiency.
- **Returns**: `Promise<void>` - A promise that resolves when the file has been successfully stored. - **Returns**: `Promise<void>` - A promise that resolves when the file has been successfully stored.
--- ---
@@ -109,9 +105,9 @@ Stores a file at the specified path. If a file already exists at that path, it w
Retrieves a file from the specified path as a readable stream. Retrieves a file from the specified path as a readable stream.
- **`path: string`**: The unique identifier of the file to retrieve. - **`path: string`**: The unique identifier of the file to retrieve.
- **Returns**: `Promise<NodeJS.ReadableStream>` - A promise that resolves with a readable stream of the file's content. - **Returns**: `Promise<NodeJS.ReadableStream>` - A promise that resolves with a readable stream of the file's content.
- **Throws**: An `Error` if the file is not found at the specified path. - **Throws**: An `Error` if the file is not found at the specified path.
--- ---
@@ -119,8 +115,8 @@ Retrieves a file from the specified path as a readable stream.
Deletes a file from the storage backend. Deletes a file from the storage backend.
- **`path: string`**: The unique identifier of the file to delete. - **`path: string`**: The unique identifier of the file to delete.
- **Returns**: `Promise<void>` - A promise that resolves when the file is deleted. If the file does not exist, the promise will still resolve successfully without throwing an error. - **Returns**: `Promise<void>` - A promise that resolves when the file is deleted. If the file does not exist, the promise will still resolve successfully without throwing an error.
--- ---
@@ -128,5 +124,5 @@ Deletes a file from the storage backend.
Checks for the existence of a file. Checks for the existence of a file.
- **`path: string`**: The unique identifier of the file to check. - **`path: string`**: The unique identifier of the file to check.
- **Returns**: `Promise<boolean>` - A promise that resolves with `true` if the file exists, and `false` otherwise. - **Returns**: `Promise<boolean>` - A promise that resolves with `true` if the file exists, and `false` otherwise.

4
docs/user-guides/email-providers/eml.md
View File

@@ -6,8 +6,8 @@ OpenArchiver allows you to import EML files from a zip archive. This is useful f
To ensure a successful import, you should compress your .eml files to one zip file according to the following guidelines: To ensure a successful import, you should compress your .eml files to one zip file according to the following guidelines:
- **Structure:** The zip file can contain any number of `.eml` files, organized in any folder structure. The folder structure will be preserved in OpenArchiver, so you can use it to organize your emails. - **Structure:** The zip file can contain any number of `.eml` files, organized in any folder structure. The folder structure will be preserved in OpenArchiver, so you can use it to organize your emails.
- **Compression:** The zip file should be compressed using standard zip compression. - **Compression:** The zip file should be compressed using standard zip compression.
Here's an example of a valid folder structure: Here's an example of a valid folder structure:

82
docs/user-guides/email-providers/google-workspace.md
View File

@@ -6,8 +6,8 @@ The connection uses a **Google Cloud Service Account** with **Domain-Wide Delega
## Prerequisites ## Prerequisites
- You must have **Super Administrator** privileges in your Google Workspace account. - You must have **Super Administrator** privileges in your Google Workspace account.
- You must have access to the **Google Cloud Console** associated with your organization. - You must have access to the **Google Cloud Console** associated with your organization.
## Setup Overview ## Setup Overview
@@ -24,30 +24,27 @@ The setup process involves three main parts:
In this part, you will create a service account and enable the APIs it needs to function. In this part, you will create a service account and enable the APIs it needs to function.
1. **Create a Google Cloud Project:** 1. **Create a Google Cloud Project:**
- Go to the [Google Cloud Console](https://console.cloud.google.com/).
- Go to the [Google Cloud Console](https://console.cloud.google.com/). - If you don't already have one, create a new project for the archiving service (e.g., "Email Archiver").
- If you don't already have one, create a new project for the archiving service (e.g., "Email Archiver").
2. **Enable Required APIs:** 2. **Enable Required APIs:**
- In your selected project, navigate to the **"APIs & Services" > "Library"** section.
- In your selected project, navigate to the **"APIs & Services" > "Library"** section. - Search for and enable the following two APIs:
- Search for and enable the following two APIs: - **Gmail API**
- **Gmail API** - **Admin SDK API**
- **Admin SDK API**
3. **Create a Service Account:** 3. **Create a Service Account:**
- Navigate to **"IAM & Admin" > "Service Accounts"**.
- Navigate to **"IAM & Admin" > "Service Accounts"**. - Click **"Create Service Account"**.
- Click **"Create Service Account"**. - Give the service account a name (e.g., `email-archiver-service`) and a description.
- Give the service account a name (e.g., `email-archiver-service`) and a description. - Click **"Create and Continue"**. You do not need to grant this service account any roles on the project. Click **"Done"**.
- Click **"Create and Continue"**. You do not need to grant this service account any roles on the project. Click **"Done"**.
4. **Generate a JSON Key:** 4. **Generate a JSON Key:**
- Find the service account you just created in the list. - Find the service account you just created in the list.
- Click the three-dot menu under **"Actions"** and select **"Manage keys"**. - Click the three-dot menu under **"Actions"** and select **"Manage keys"**.
- Click **"Add Key"** > **"Create new key"**. - Click **"Add Key"** > **"Create new key"**.
- Select **JSON** as the key type and click **"Create"**. - Select **JSON** as the key type and click **"Create"**.
- A JSON file will be downloaded to your computer. **Keep this file secure, as it contains private credentials.** You will need the contents of this file in Part 3. - A JSON file will be downloaded to your computer. **Keep this file secure, as it contains private credentials.** You will need the contents of this file in Part 3.
### Troubleshooting ### Troubleshooting
@@ -60,14 +57,14 @@ To resolve this, you must have **Organization Administrator** permissions.
1. **Navigate to your Organization:** In the Google Cloud Console, use the project selector at the top of the page to select your organization node (it usually has a building icon). 1. **Navigate to your Organization:** In the Google Cloud Console, use the project selector at the top of the page to select your organization node (it usually has a building icon).
2. **Go to IAM:** From the navigation menu, select **"IAM & Admin" > "IAM"**. 2. **Go to IAM:** From the navigation menu, select **"IAM & Admin" > "IAM"**.
3. **Edit Your Permissions:** Find your user account in the list and click the pencil icon to edit roles. Add the following two roles: 3. **Edit Your Permissions:** Find your user account in the list and click the pencil icon to edit roles. Add the following two roles:
- `Organization Policy Administrator` - `Organization Policy Administrator`
- `Organization Administrator` - `Organization Administrator`
_Note: These roles are only available at the organization level, not the project level._ _Note: These roles are only available at the organization level, not the project level._
4. **Modify the Policy:** 4. **Modify the Policy:**
- Navigate to **"IAM & Admin" > "Organization Policies"**. - Navigate to **"IAM & Admin" > "Organization Policies"**.
- In the filter box, search for the policy **"iam.disableServiceAccountKeyCreation"**. - In the filter box, search for the policy **"iam.disableServiceAccountKeyCreation"**.
- Click on the policy to edit it. - Click on the policy to edit it.
- You can either disable the policy entirely (if your security rules permit) or add a rule to exclude the specific project you are using for the archiver from this policy. - You can either disable the policy entirely (if your security rules permit) or add a rule to exclude the specific project you are using for the archiver from this policy.
5. **Retry Key Creation:** Once the policy is updated, return to your project and you should be able to generate the JSON key as described in Part 1. 5. **Retry Key Creation:** Once the policy is updated, return to your project and you should be able to generate the JSON key as described in Part 1.
--- ---
@@ -77,25 +74,23 @@ To resolve this, you must have **Organization Administrator** permissions.
Now, you will authorize the service account you created to access data from your Google Workspace. Now, you will authorize the service account you created to access data from your Google Workspace.
1. **Get the Service Account's Client ID:** 1. **Get the Service Account's Client ID:**
- Go back to the list of service accounts in the Google Cloud Console.
- Go back to the list of service accounts in the Google Cloud Console. - Click on the service account you created.
- Click on the service account you created. - Under the **"Details"** tab, find and copy the **Unique ID** (this is the Client ID).
- Under the **"Details"** tab, find and copy the **Unique ID** (this is the Client ID).
2. **Authorize the Client in Google Workspace:** 2. **Authorize the Client in Google Workspace:**
- Go to your **Google Workspace Admin Console** at [admin.google.com](https://admin.google.com).
- Go to your **Google Workspace Admin Console** at [admin.google.com](https://admin.google.com). - Navigate to **Security > Access and data control > API controls**.
- Navigate to **Security > Access and data control > API controls**. - Under the "Domain-wide Delegation" section, click **"Manage Domain-wide Delegation"**.
- Under the "Domain-wide Delegation" section, click **"Manage Domain-wide Delegation"**. - Click **"Add new"**.
- Click **"Add new"**.
3. **Enter Client Details and Scopes:** 3. **Enter Client Details and Scopes:**
- In the **Client ID** field, paste the **Unique ID** you copied from the service account. - In the **Client ID** field, paste the **Unique ID** you copied from the service account.
- In the **OAuth scopes** field, paste the following two scopes exactly as they appear, separated by a comma: - In the **OAuth scopes** field, paste the following two scopes exactly as they appear, separated by a comma:
``` ```
https://www.googleapis.com/auth/admin.directory.user.readonly,https://www.googleapis.com/auth/gmail.readonly https://www.googleapis.com/auth/admin.directory.user.readonly,https://www.googleapis.com/auth/gmail.readonly
``` ```
- Click **"Authorize"**. - Click **"Authorize"**.
The service account is now permitted to list users and read their email data across your domain. The service account is now permitted to list users and read their email data across your domain.
@@ -112,11 +107,10 @@ Finally, you will provide the generated credentials to the application.
Click the **"Create New"** button. Click the **"Create New"** button.
3. **Fill in the Configuration Details:** 3. **Fill in the Configuration Details:**
- **Name:** Give the source a name (e.g., "Google Workspace Archive").
- **Name:** Give the source a name (e.g., "Google Workspace Archive"). - **Provider:** Select **"Google Workspace"** from the dropdown.
- **Provider:** Select **"Google Workspace"** from the dropdown. - **Service Account Key (JSON):** Open the JSON file you downloaded in Part 1. Copy the entire content of the file and paste it into this text area.
- **Service Account Key (JSON):** Open the JSON file you downloaded in Part 1. Copy the entire content of the file and paste it into this text area. - **Impersonated Admin Email:** Enter the email address of a Super Administrator in your Google Workspace (e.g., `admin@your-domain.com`). The service will use this user's authority to discover all other users.
- **Impersonated Admin Email:** Enter the email address of a Super Administrator in your Google Workspace (e.g., `admin@your-domain.com`). The service will use this user's authority to discover all other users.
4. **Save Changes:** 4. **Save Changes:**
Click **"Save changes"**. Click **"Save changes"**.

31
docs/user-guides/email-providers/imap.md
View File

@@ -12,18 +12,17 @@ This guide will walk you through connecting a standard IMAP email account as an
3. **Fill in the Configuration Details:** 3. **Fill in the Configuration Details:**
You will see a form with several fields. Here is how to fill them out for an IMAP connection: You will see a form with several fields. Here is how to fill them out for an IMAP connection:
- **Name:** Give your ingestion source a descriptive name that you will easily recognize, such as "Work Email (IMAP)" or "Personal Gmail".
- **Name:** Give your ingestion source a descriptive name that you will easily recognize, such as "Work Email (IMAP)" or "Personal Gmail". - **Provider:** From the dropdown menu, select **"Generic IMAP"**. This will reveal the specific fields required for an IMAP connection.
- **Provider:** From the dropdown menu, select **"Generic IMAP"**. This will reveal the specific fields required for an IMAP connection. - **Host:** Enter the server address for your email provider's IMAP service. This often looks like `imap.your-provider.com` or `mail.your-domain.com`.
- **Host:** Enter the server address for your email provider's IMAP service. This often looks like `imap.your-provider.com` or `mail.your-domain.com`. - **Port:** Enter the port number for the IMAP server. For a secure connection (which is strongly recommended), this is typically `993`.
- **Port:** Enter the port number for the IMAP server. For a secure connection (which is strongly recommended), this is typically `993`. - **Username:** Enter the full email address or username you use to log in to your email account.
- **Username:** Enter the full email address or username you use to log in to your email account. - **Password:** Enter the password for your email account.
- **Password:** Enter the password for your email account.
4. **Save Changes:** 4. **Save Changes:**
Once you have filled in all the details, click the **"Save changes"** button. Once you have filled in all the details, click the **"Save changes"** button.
@@ -41,9 +40,9 @@ Please consult your email provider's documentation to see if they support app pa
1. **Enable 2-Step Verification:** You must have 2-Step Verification turned on for your Google Account. 1. **Enable 2-Step Verification:** You must have 2-Step Verification turned on for your Google Account.
2. **Go to App Passwords:** Visit [myaccount.google.com/apppasswords](https://myaccount.google.com/apppasswords). You may be asked to sign in again. 2. **Go to App Passwords:** Visit [myaccount.google.com/apppasswords](https://myaccount.google.com/apppasswords). You may be asked to sign in again.
3. **Create the Password:** 3. **Create the Password:**
- At the bottom, click **"Select app"** and choose **"Other (Custom name)"**. - At the bottom, click **"Select app"** and choose **"Other (Custom name)"**.
- Give it a name you'll recognize, like "OpenArchiver". - Give it a name you'll recognize, like "OpenArchiver".
- Click **"Generate"**. - Click **"Generate"**.
4. **Use the Password:** A 16-digit password will be displayed. Copy this password and paste it into the **Password** field in the OpenArchiver ingestion source form. 4. **Use the Password:** A 16-digit password will be displayed. Copy this password and paste it into the **Password** field in the OpenArchiver ingestion source form.
### How to Obtain an App Password for Outlook/Microsoft Accounts ### How to Obtain an App Password for Outlook/Microsoft Accounts
@@ -51,17 +50,17 @@ Please consult your email provider's documentation to see if they support app pa
1. **Enable Two-Step Verification:** You must have two-step verification enabled for your Microsoft account. 1. **Enable Two-Step Verification:** You must have two-step verification enabled for your Microsoft account.
2. **Go to Security Options:** Sign in to your Microsoft account and navigate to the [Advanced security options](https://account.live.com/proofs/manage/additional). 2. **Go to Security Options:** Sign in to your Microsoft account and navigate to the [Advanced security options](https://account.live.com/proofs/manage/additional).
3. **Create a New App Password:** 3. **Create a New App Password:**
- Scroll down to the **"App passwords"** section. - Scroll down to the **"App passwords"** section.
- Click **"Create a new app password"**. - Click **"Create a new app password"**.
4. **Use the Password:** A new password will be generated. Use this password in the **Password** field in the OpenArchiver ingestion source form. 4. **Use the Password:** A new password will be generated. Use this password in the **Password** field in the OpenArchiver ingestion source form.
## What Happens Next? ## What Happens Next?
After you save the connection, the system will attempt to connect to the IMAP server. The status of the ingestion source will update to reflect its current state: After you save the connection, the system will attempt to connect to the IMAP server. The status of the ingestion source will update to reflect its current state:
- **Importing:** The system is performing the initial, one-time import of all emails from your `INBOX`. This may take a while depending on the size of your mailbox. - **Importing:** The system is performing the initial, one-time import of all emails from your `INBOX`. This may take a while depending on the size of your mailbox.
- **Active:** The initial import is complete, and the system will now periodically check for and archive new emails. - **Active:** The initial import is complete, and the system will now periodically check for and archive new emails.
- **Paused:** The connection is valid, but the system will not check for new emails until you resume it. - **Paused:** The connection is valid, but the system will not check for new emails until you resume it.
- **Error:** The system was unable to connect using the provided credentials. Please double-check your Host, Port, Username, and Password and try again. - **Error:** The system was unable to connect using the provided credentials. Please double-check your Host, Port, Username, and Password and try again.
You can view, edit, pause, or manually sync any of your ingestion sources from the main table on the **Ingestions** page. You can view, edit, pause, or manually sync any of your ingestion sources from the main table on the **Ingestions** page.

10
docs/user-guides/email-providers/index.md
View File

@@ -4,8 +4,8 @@ Open Archiver can connect to a variety of email sources to ingest and archive yo
Choose your provider from the list below to get started: Choose your provider from the list below to get started:
- [Google Workspace](./google-workspace.md) - [Google Workspace](./google-workspace.md)
- [Microsoft 365](./microsoft-365.md) - [Microsoft 365](./microsoft-365.md)
- [Generic IMAP Server](./imap.md) - [Generic IMAP Server](./imap.md)
- [EML Import](./eml.md) - [EML Import](./eml.md)
- [PST Import](./pst.md) - [PST Import](./pst.md)

27
docs/user-guides/email-providers/microsoft-365.md
View File

@@ -6,7 +6,7 @@ The connection uses the **Microsoft Graph API** and an **App Registration** in M
## Prerequisites ## Prerequisites
- You must have one of the following administrator roles in your Microsoft 365 tenant: **Global Administrator**, **Application Administrator**, or **Cloud Application Administrator**. - You must have one of the following administrator roles in your Microsoft 365 tenant: **Global Administrator**, **Application Administrator**, or **Cloud Application Administrator**.
## Setup Overview ## Setup Overview
@@ -27,9 +27,9 @@ First, you will create an "App registration," which acts as an identity for the
2. In the left-hand navigation pane, go to **Identity > Applications > App registrations**. 2. In the left-hand navigation pane, go to **Identity > Applications > App registrations**.
3. Click the **+ New registration** button at the top of the page. 3. Click the **+ New registration** button at the top of the page.
4. On the "Register an application" screen: 4. On the "Register an application" screen:
- **Name:** Give the application a descriptive name you will recognize, such as `OpenArchiver Service`. - **Name:** Give the application a descriptive name you will recognize, such as `OpenArchiver Service`.
- **Supported account types:** Select **"Accounts in this organizational directory only (Default Directory only - Single tenant)"**. This is the most secure option. - **Supported account types:** Select **"Accounts in this organizational directory only (Default Directory only - Single tenant)"**. This is the most secure option.
- **Redirect URI (optional):** You can leave this blank. - **Redirect URI (optional):** You can leave this blank.
5. Click the **Register** button. You will be taken to the application's main "Overview" page. 5. Click the **Register** button. You will be taken to the application's main "Overview" page.
--- ---
@@ -43,8 +43,8 @@ Next, you must grant the application the specific permissions required to read u
3. In the "Request API permissions" pane, select **Microsoft Graph**. 3. In the "Request API permissions" pane, select **Microsoft Graph**.
4. Select **Application permissions**. This is critical as it allows the service to run in the background without a user being signed in. 4. Select **Application permissions**. This is critical as it allows the service to run in the background without a user being signed in.
5. In the "Select permissions" search box, find and check the boxes for the following two permissions: 5. In the "Select permissions" search box, find and check the boxes for the following two permissions:
- `Mail.Read` - `Mail.Read`
- `User.Read.All` - `User.Read.All`
6. Click the **Add permissions** button at the bottom. 6. Click the **Add permissions** button at the bottom.
7. **Crucial Final Step:** You will now see the permissions in your list with a warning status. You must grant consent on behalf of your organization. Click the **"Grant admin consent for [Your Organization's Name]"** button located above the permissions table. Click **Yes** in the confirmation dialog. The status for both permissions should now show a green checkmark. 7. **Crucial Final Step:** You will now see the permissions in your list with a warning status. You must grant consent on behalf of your organization. Click the **"Grant admin consent for [Your Organization's Name]"** button located above the permissions table. Click **Yes** in the confirmation dialog. The status for both permissions should now show a green checkmark.
@@ -57,8 +57,8 @@ The client secret is a password that the archiving service will use to authentic
1. In your application's menu, navigate to **Certificates & secrets**. 1. In your application's menu, navigate to **Certificates & secrets**.
2. Select the **Client secrets** tab and click **+ New client secret**. 2. Select the **Client secrets** tab and click **+ New client secret**.
3. In the pane that appears: 3. In the pane that appears:
- **Description:** Enter a clear description, such as `OpenArchiver Key`. - **Description:** Enter a clear description, such as `OpenArchiver Key`.
- **Expires:** Select an expiry duration. We recommend **12 or 24 months**. Set a calendar reminder to renew it before it expires to prevent service interruption. - **Expires:** Select an expiry duration. We recommend **12 or 24 months**. Set a calendar reminder to renew it before it expires to prevent service interruption.
4. Click **Add**. 4. Click **Add**.
5. **IMMEDIATELY COPY THE SECRET:** The secret is now visible in the **"Value"** column. This is the only time it will be fully displayed. Copy this value now and store it in a secure password manager before navigating away. If you lose it, you must create a new one. 5. **IMMEDIATELY COPY THE SECRET:** The secret is now visible in the **"Value"** column. This is the only time it will be fully displayed. Copy this value now and store it in a secure password manager before navigating away. If you lose it, you must create a new one.
@@ -75,12 +75,11 @@ You now have the three pieces of information required to configure the connectio
Click the **"Create New"** button. Click the **"Create New"** button.
3. **Fill in the Configuration Details:** 3. **Fill in the Configuration Details:**
- **Name:** Give the source a name (e.g., "Microsoft 365 Archive").
- **Name:** Give the source a name (e.g., "Microsoft 365 Archive"). - **Provider:** Select **"Microsoft 365"** from the dropdown.
- **Provider:** Select **"Microsoft 365"** from the dropdown. - **Application (Client) ID:** Go to the **Overview** page of your app registration in the Entra admin center and copy this value.
- **Application (Client) ID:** Go to the **Overview** page of your app registration in the Entra admin center and copy this value. - **Directory (Tenant) ID:** This value is also on the **Overview** page.
- **Directory (Tenant) ID:** This value is also on the **Overview** page. - **Client Secret Value:** Paste the secret **Value** (not the Secret ID) that you copied and saved in the previous step.
- **Client Secret Value:** Paste the secret **Value** (not the Secret ID) that you copied and saved in the previous step.
4. **Save Changes:** 4. **Save Changes:**
Click **"Save changes"**. Click **"Save changes"**.

4
docs/user-guides/email-providers/pst.md
View File

@@ -6,8 +6,8 @@ OpenArchiver allows you to import PST files. This is useful for importing emails
To ensure a successful import, you should prepare your PST file according to the following guidelines: To ensure a successful import, you should prepare your PST file according to the following guidelines:
- **Structure:** The PST file can contain any number of emails, organized in any folder structure. The folder structure will be preserved in OpenArchiver, so you can use it to organize your emails. - **Structure:** The PST file can contain any number of emails, organized in any folder structure. The folder structure will be preserved in OpenArchiver, so you can use it to organize your emails.
- **Password Protection:** OpenArchiver does not support password-protected PST files. Please remove the password from your PST file before importing it. - **Password Protection:** OpenArchiver does not support password-protected PST files. Please remove the password from your PST file before importing it.
## Creating a PST Ingestion Source ## Creating a PST Ingestion Source

33
docs/user-guides/installation.md
View File

@@ -4,9 +4,9 @@ This guide will walk you through setting up Open Archiver using Docker Compose.
## Prerequisites ## Prerequisites
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed on your server or local machine. - [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed on your server or local machine.
- A server or local machine with at least 4GB of RAM (2GB of RAM if you use external Postgres, Redis (Valkey) and Meilisearch instances). - A server or local machine with at least 4GB of RAM (2GB of RAM if you use external Postgres, Redis (Valkey) and Meilisearch instances).
- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed on your server or local machine. - [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed on your server or local machine.
## 1. Clone the Repository ## 1. Clone the Repository
@@ -33,11 +33,11 @@ Now, open the `.env` file in a text editor and customize the settings.
You must change the following placeholder values to secure your instance: You must change the following placeholder values to secure your instance:
- `POSTGRES_PASSWORD`: A strong, unique password for the database. - `POSTGRES_PASSWORD`: A strong, unique password for the database.
- `REDIS_PASSWORD`: A strong, unique password for the Valkey/Redis service. - `REDIS_PASSWORD`: A strong, unique password for the Valkey/Redis service.
- `MEILI_MASTER_KEY`: A complex key for Meilisearch. - `MEILI_MASTER_KEY`: A complex key for Meilisearch.
- `JWT_SECRET`: A long, random string for signing authentication tokens. - `JWT_SECRET`: A long, random string for signing authentication tokens.
- `ENCRYPTION_KEY`: A 32-byte hex string for encrypting sensitive data in the database. You can generate one with the following command: - `ENCRYPTION_KEY`: A 32-byte hex string for encrypting sensitive data in the database. You can generate one with the following command:
```bash ```bash
openssl rand -hex 32 openssl rand -hex 32
``` ```
@@ -122,9 +122,9 @@ docker compose up -d
This command will: This command will:
- Pull the required Docker images for the frontend, backend, database, and other services. - Pull the required Docker images for the frontend, backend, database, and other services.
- Create and start the containers in the background (`-d` flag). - Create and start the containers in the background (`-d` flag).
- Create the persistent volumes for your data. - Create the persistent volumes for your data.
You can check the status of the running containers with: You can check the status of the running containers with:
@@ -142,9 +142,9 @@ You can log in with the `ADMIN_EMAIL` and `ADMIN_PASSWORD` you configured in you
After successfully deploying and logging into Open Archiver, the next step is to configure your ingestion sources to start archiving emails. After successfully deploying and logging into Open Archiver, the next step is to configure your ingestion sources to start archiving emails.
- [Connecting to Google Workspace](./email-providers/google-workspace.md) - [Connecting to Google Workspace](./email-providers/google-workspace.md)
- [Connecting to Microsoft 365](./email-providers/microsoft-365.md) - [Connecting to Microsoft 365](./email-providers/microsoft-365.md)
- [Connecting to a Generic IMAP Server](./email-providers/imap.md) - [Connecting to a Generic IMAP Server](./email-providers/imap.md)
## Updating Your Installation ## Updating Your Installation
@@ -174,9 +174,8 @@ To do this, you will need to make a small modification to your `docker-compose.y
2. **Remove all `networks` sections** from the file. This includes the network configuration for each service and the top-level network definition. 2. **Remove all `networks` sections** from the file. This includes the network configuration for each service and the top-level network definition.
Specifically, you need to remove: Specifically, you need to remove:
- The `networks: - open-archiver-net` lines from the `open-archiver`, `postgres`, `valkey`, and `meilisearch` services.
- The `networks: - open-archiver-net` lines from the `open-archiver`, `postgres`, `valkey`, and `meilisearch` services. - The entire `networks:` block at the end of the file.
- The entire `networks:` block at the end of the file.
Here is an example of what to remove from a service: Here is an example of what to remove from a service:

4
package.json
View File

@@ -14,7 +14,7 @@
"docs:dev": "vitepress dev docs --port 3009", "docs:dev": "vitepress dev docs --port 3009",
"docs:build": "vitepress build docs", "docs:build": "vitepress build docs",
"docs:preview": "vitepress preview docs", "docs:preview": "vitepress preview docs",
"format": "prettier --write .", "format": "prettier --write .",
"lint": "prettier --check ." "lint": "prettier --check ."
}, },
"dependencies": { "dependencies": {
@@ -23,7 +23,7 @@
}, },
"devDependencies": { "devDependencies": {
"prettier": "^3.6.2", "prettier": "^3.6.2",
"prettier-plugin-svelte": "^3.4.0", "prettier-plugin-svelte": "^3.4.0",
"prettier-plugin-tailwindcss": "^0.6.14", "prettier-plugin-tailwindcss": "^0.6.14",
"typescript": "5.8.3", "typescript": "5.8.3",
"vitepress": "^1.6.4" "vitepress": "^1.6.4"

18
packages/backend/drizzle.config.ts
View File

@@ -4,16 +4,16 @@ import { config } from 'dotenv';
config(); config();
if (!process.env.DATABASE_URL) { if (!process.env.DATABASE_URL) {
throw new Error('DATABASE_URL is not set in the .env file'); throw new Error('DATABASE_URL is not set in the .env file');
} }
export default defineConfig({ export default defineConfig({
schema: './src/database/schema.ts', schema: './src/database/schema.ts',
out: './src/database/migrations', out: './src/database/migrations',
dialect: 'postgresql', dialect: 'postgresql',
dbCredentials: { dbCredentials: {
url: process.env.DATABASE_URL, url: process.env.DATABASE_URL,
}, },
verbose: true, verbose: true,
strict: true, strict: true,
}); });

146
packages/backend/package.json
View File

@@ -1,75 +1,75 @@
{ {
"name": "@open-archiver/backend", "name": "@open-archiver/backend",
"version": "0.1.0", "version": "0.1.0",
"private": true, "private": true,
"main": "dist/index.js", "main": "dist/index.js",
"scripts": { "scripts": {
"dev": "ts-node-dev --respawn --transpile-only src/index.ts ", "dev": "ts-node-dev --respawn --transpile-only src/index.ts ",
"build": "tsc", "build": "tsc",
"start": "node dist/index.js", "start": "node dist/index.js",
"start:ingestion-worker": "node dist/workers/ingestion.worker.js", "start:ingestion-worker": "node dist/workers/ingestion.worker.js",
"start:indexing-worker": "node dist/workers/indexing.worker.js", "start:indexing-worker": "node dist/workers/indexing.worker.js",
"start:sync-scheduler": "node dist/jobs/schedulers/sync-scheduler.js", "start:sync-scheduler": "node dist/jobs/schedulers/sync-scheduler.js",
"start:ingestion-worker:dev": "ts-node-dev --respawn --transpile-only src/workers/ingestion.worker.ts", "start:ingestion-worker:dev": "ts-node-dev --respawn --transpile-only src/workers/ingestion.worker.ts",
"start:indexing-worker:dev": "ts-node-dev --respawn --transpile-only src/workers/indexing.worker.ts", "start:indexing-worker:dev": "ts-node-dev --respawn --transpile-only src/workers/indexing.worker.ts",
"start:sync-scheduler:dev": "ts-node-dev --respawn --transpile-only src/jobs/schedulers/sync-scheduler.ts", "start:sync-scheduler:dev": "ts-node-dev --respawn --transpile-only src/jobs/schedulers/sync-scheduler.ts",
"db:generate": "drizzle-kit generate --config=drizzle.config.ts", "db:generate": "drizzle-kit generate --config=drizzle.config.ts",
"db:push": "drizzle-kit push --config=drizzle.config.ts", "db:push": "drizzle-kit push --config=drizzle.config.ts",
"db:migrate": "node dist/database/migrate.js", "db:migrate": "node dist/database/migrate.js",
"db:migrate:dev": "ts-node-dev src/database/migrate.ts" "db:migrate:dev": "ts-node-dev src/database/migrate.ts"
}, },
"dependencies": { "dependencies": {
"@aws-sdk/client-s3": "^3.844.0", "@aws-sdk/client-s3": "^3.844.0",
"@aws-sdk/lib-storage": "^3.844.0", "@aws-sdk/lib-storage": "^3.844.0",
"@azure/msal-node": "^3.6.3", "@azure/msal-node": "^3.6.3",
"@microsoft/microsoft-graph-client": "^3.0.7", "@microsoft/microsoft-graph-client": "^3.0.7",
"@open-archiver/types": "workspace:*", "@open-archiver/types": "workspace:*",
"archiver": "^7.0.1", "archiver": "^7.0.1",
"axios": "^1.10.0", "axios": "^1.10.0",
"bcryptjs": "^3.0.2", "bcryptjs": "^3.0.2",
"bullmq": "^5.56.3", "bullmq": "^5.56.3",
"busboy": "^1.6.0", "busboy": "^1.6.0",
"cross-fetch": "^4.1.0", "cross-fetch": "^4.1.0",
"deepmerge-ts": "^7.1.5", "deepmerge-ts": "^7.1.5",
"dotenv": "^17.2.0", "dotenv": "^17.2.0",
"drizzle-kit": "^0.31.4", "drizzle-kit": "^0.31.4",
"drizzle-orm": "^0.44.2", "drizzle-orm": "^0.44.2",
"express": "^5.1.0", "express": "^5.1.0",
"express-rate-limit": "^8.0.1", "express-rate-limit": "^8.0.1",
"express-validator": "^7.2.1", "express-validator": "^7.2.1",
"google-auth-library": "^10.1.0", "google-auth-library": "^10.1.0",
"googleapis": "^152.0.0", "googleapis": "^152.0.0",
"imapflow": "^1.0.191", "imapflow": "^1.0.191",
"jose": "^6.0.11", "jose": "^6.0.11",
"mailparser": "^3.7.4", "mailparser": "^3.7.4",
"mammoth": "^1.9.1", "mammoth": "^1.9.1",
"meilisearch": "^0.51.0", "meilisearch": "^0.51.0",
"multer": "^2.0.2", "multer": "^2.0.2",
"pdf2json": "^3.1.6", "pdf2json": "^3.1.6",
"pg": "^8.16.3", "pg": "^8.16.3",
"pino": "^9.7.0", "pino": "^9.7.0",
"pino-pretty": "^13.0.0", "pino-pretty": "^13.0.0",
"postgres": "^3.4.7", "postgres": "^3.4.7",
"pst-extractor": "^1.11.0", "pst-extractor": "^1.11.0",
"reflect-metadata": "^0.2.2", "reflect-metadata": "^0.2.2",
"sqlite3": "^5.1.7", "sqlite3": "^5.1.7",
"tsconfig-paths": "^4.2.0", "tsconfig-paths": "^4.2.0",
"xlsx": "^0.18.5", "xlsx": "^0.18.5",
"yauzl": "^3.2.0" "yauzl": "^3.2.0"
}, },
"devDependencies": { "devDependencies": {
"@bull-board/api": "^6.11.0", "@bull-board/api": "^6.11.0",
"@bull-board/express": "^6.11.0", "@bull-board/express": "^6.11.0",
"@types/archiver": "^6.0.3", "@types/archiver": "^6.0.3",
"@types/busboy": "^1.5.4", "@types/busboy": "^1.5.4",
"@types/express": "^5.0.3", "@types/express": "^5.0.3",
"@types/mailparser": "^3.4.6", "@types/mailparser": "^3.4.6",
"@types/microsoft-graph": "^2.40.1", "@types/microsoft-graph": "^2.40.1",
"@types/multer": "^2.0.0", "@types/multer": "^2.0.0",
"@types/node": "^24.0.12", "@types/node": "^24.0.12",
"@types/yauzl": "^2.10.3", "@types/yauzl": "^2.10.3",
"bull-board": "^2.1.3", "bull-board": "^2.1.3",
"ts-node-dev": "^2.0.0", "ts-node-dev": "^2.0.0",
"typescript": "^5.8.3" "typescript": "^5.8.3"
} }
} }

98
packages/backend/src/api/controllers/archived-email.controller.ts
View File

@@ -3,57 +3,55 @@ import { ArchivedEmailService } from '../../services/ArchivedEmailService';
import { config } from '../../config'; import { config } from '../../config';
export class ArchivedEmailController { export class ArchivedEmailController {
public getArchivedEmails = async (req: Request, res: Response): Promise<Response> => { public getArchivedEmails = async (req: Request, res: Response): Promise<Response> => {
try { try {
const { ingestionSourceId } = req.params; const { ingestionSourceId } = req.params;
const page = parseInt(req.query.page as string, 10) || 1; const page = parseInt(req.query.page as string, 10) || 1;
const limit = parseInt(req.query.limit as string, 10) || 10; const limit = parseInt(req.query.limit as string, 10) || 10;
const result = await ArchivedEmailService.getArchivedEmails( const result = await ArchivedEmailService.getArchivedEmails(
ingestionSourceId, ingestionSourceId,
page, page,
limit limit
); );
return res.status(200).json(result); return res.status(200).json(result);
} catch (error) { } catch (error) {
console.error('Get archived emails error:', error); console.error('Get archived emails error:', error);
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
public getArchivedEmailById = async (req: Request, res: Response): Promise<Response> => { public getArchivedEmailById = async (req: Request, res: Response): Promise<Response> => {
try { try {
const { id } = req.params; const { id } = req.params;
const email = await ArchivedEmailService.getArchivedEmailById(id); const email = await ArchivedEmailService.getArchivedEmailById(id);
if (!email) { if (!email) {
return res.status(404).json({ message: 'Archived email not found' }); return res.status(404).json({ message: 'Archived email not found' });
} }
return res.status(200).json(email); return res.status(200).json(email);
} catch (error) { } catch (error) {
console.error(`Get archived email by id ${req.params.id} error:`, error); console.error(`Get archived email by id ${req.params.id} error:`, error);
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
public deleteArchivedEmail = async (req: Request, res: Response): Promise<Response> => { public deleteArchivedEmail = async (req: Request, res: Response): Promise<Response> => {
if (config.app.isDemo) { if (config.app.isDemo) {
return res return res.status(403).json({ message: 'This operation is not allowed in demo mode.' });
.status(403) }
.json({ message: 'This operation is not allowed in demo mode.' }); try {
} const { id } = req.params;
try { await ArchivedEmailService.deleteArchivedEmail(id);
const { id } = req.params; return res.status(204).send();
await ArchivedEmailService.deleteArchivedEmail(id); } catch (error) {
return res.status(204).send(); console.error(`Delete archived email ${req.params.id} error:`, error);
} catch (error) { if (error instanceof Error) {
console.error(`Delete archived email ${req.params.id} error:`, error); if (error.message === 'Archived email not found') {
if (error instanceof Error) { return res.status(404).json({ message: error.message });
if (error.message === 'Archived email not found') { }
return res.status(404).json({ message: error.message }); return res.status(500).json({ message: error.message });
} }
return res.status(500).json({ message: error.message }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
return res.status(500).json({ message: 'An internal server error occurred' }); };
}
};
} }

150
packages/backend/src/api/controllers/auth.controller.ts
View File

@@ -6,88 +6,94 @@ import * as schema from '../../database/schema';
import { sql } from 'drizzle-orm'; import { sql } from 'drizzle-orm';
import 'dotenv/config'; import 'dotenv/config';
export class AuthController { export class AuthController {
#authService: AuthService; #authService: AuthService;
#userService: UserService; #userService: UserService;
constructor(authService: AuthService, userService: UserService) { constructor(authService: AuthService, userService: UserService) {
this.#authService = authService; this.#authService = authService;
this.#userService = userService; this.#userService = userService;
} }
/** /**
* Only used for setting up the instance, should only be displayed once upon instance set up. * Only used for setting up the instance, should only be displayed once upon instance set up.
* @param req * @param req
* @param res * @param res
* @returns * @returns
*/ */
public setup = async (req: Request, res: Response): Promise<Response> => { public setup = async (req: Request, res: Response): Promise<Response> => {
const { email, password, first_name, last_name } = req.body; const { email, password, first_name, last_name } = req.body;
if (!email || !password || !first_name || !last_name) { if (!email || !password || !first_name || !last_name) {
return res.status(400).json({ message: 'Email, password, and name are required' }); return res.status(400).json({ message: 'Email, password, and name are required' });
} }
try { try {
const userCountResult = await db.select({ count: sql<number>`count(*)` }).from(schema.users); const userCountResult = await db
const userCount = Number(userCountResult[0].count); .select({ count: sql<number>`count(*)` })
.from(schema.users);
const userCount = Number(userCountResult[0].count);
if (userCount > 0) { if (userCount > 0) {
return res.status(403).json({ message: 'Setup has already been completed.' }); return res.status(403).json({ message: 'Setup has already been completed.' });
} }
const newUser = await this.#userService.createAdminUser({ email, password, first_name, last_name }, true); const newUser = await this.#userService.createAdminUser(
const result = await this.#authService.login(email, password); { email, password, first_name, last_name },
return res.status(201).json(result); true
} catch (error) { );
console.error('Setup error:', error); const result = await this.#authService.login(email, password);
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(201).json(result);
} } catch (error) {
}; console.error('Setup error:', error);
return res.status(500).json({ message: 'An internal server error occurred' });
}
};
public login = async (req: Request, res: Response): Promise<Response> => { public login = async (req: Request, res: Response): Promise<Response> => {
const { email, password } = req.body; const { email, password } = req.body;
if (!email || !password) { if (!email || !password) {
return res.status(400).json({ message: 'Email and password are required' }); return res.status(400).json({ message: 'Email and password are required' });
} }
try { try {
const result = await this.#authService.login(email, password); const result = await this.#authService.login(email, password);
if (!result) { if (!result) {
return res.status(401).json({ message: 'Invalid credentials' }); return res.status(401).json({ message: 'Invalid credentials' });
} }
return res.status(200).json(result); return res.status(200).json(result);
} catch (error) { } catch (error) {
console.error('Login error:', error); console.error('Login error:', error);
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
public status = async (req: Request, res: Response): Promise<Response> => { public status = async (req: Request, res: Response): Promise<Response> => {
try { try {
const userCountResult = await db
.select({ count: sql<number>`count(*)` })
.from(schema.users);
const userCountResult = await db.select({ count: sql<number>`count(*)` }).from(schema.users); const userCount = Number(userCountResult[0].count);
const userCount = Number(userCountResult[0].count); const needsSetup = userCount === 0;
const needsSetup = userCount === 0; // in case user uses older version with admin user variables, we will create the admin user using those variables.
// in case user uses older version with admin user variables, we will create the admin user using those variables. if (needsSetup && process.env.ADMIN_EMAIL && process.env.ADMIN_PASSWORD) {
if (needsSetup && process.env.ADMIN_EMAIL && process.env.ADMIN_PASSWORD) { await this.#userService.createAdminUser(
await this.#userService.createAdminUser({ {
email: process.env.ADMIN_EMAIL, email: process.env.ADMIN_EMAIL,
password: process.env.ADMIN_PASSWORD, password: process.env.ADMIN_PASSWORD,
first_name: "Admin", first_name: 'Admin',
last_name: "User" last_name: 'User',
}, true); },
return res.status(200).json({ needsSetup: false }); true
} );
return res.status(200).json({ needsSetup }); return res.status(200).json({ needsSetup: false });
} catch (error) { }
console.error('Status check error:', error); return res.status(200).json({ needsSetup });
return res.status(500).json({ message: 'An internal server error occurred' }); } catch (error) {
} console.error('Status check error:', error);
}; return res.status(500).json({ message: 'An internal server error occurred' });
}
};
} }

40
packages/backend/src/api/controllers/dashboard.controller.ts
View File

@@ -2,30 +2,30 @@ import { Request, Response } from 'express';
import { dashboardService } from '../../services/DashboardService'; import { dashboardService } from '../../services/DashboardService';
class DashboardController { class DashboardController {
public async getStats(req: Request, res: Response) { public async getStats(req: Request, res: Response) {
const stats = await dashboardService.getStats(); const stats = await dashboardService.getStats();
res.json(stats); res.json(stats);
} }
public async getIngestionHistory(req: Request, res: Response) { public async getIngestionHistory(req: Request, res: Response) {
const history = await dashboardService.getIngestionHistory(); const history = await dashboardService.getIngestionHistory();
res.json(history); res.json(history);
} }
public async getIngestionSources(req: Request, res: Response) { public async getIngestionSources(req: Request, res: Response) {
const sources = await dashboardService.getIngestionSources(); const sources = await dashboardService.getIngestionSources();
res.json(sources); res.json(sources);
} }
public async getRecentSyncs(req: Request, res: Response) { public async getRecentSyncs(req: Request, res: Response) {
const syncs = await dashboardService.getRecentSyncs(); const syncs = await dashboardService.getRecentSyncs();
res.json(syncs); res.json(syncs);
} }
public async getIndexedInsights(req: Request, res: Response) { public async getIndexedInsights(req: Request, res: Response) {
const insights = await dashboardService.getIndexedInsights(); const insights = await dashboardService.getIndexedInsights();
res.json(insights); res.json(insights);
} }
} }
export const dashboardController = new DashboardController(); export const dashboardController = new DashboardController();

108
packages/backend/src/api/controllers/iam.controller.ts
View File

@@ -4,68 +4,68 @@ import { PolicyValidator } from '../../iam-policy/policy-validator';
import type { PolicyStatement } from '@open-archiver/types'; import type { PolicyStatement } from '@open-archiver/types';
export class IamController { export class IamController {
#iamService: IamService; #iamService: IamService;
constructor(iamService: IamService) { constructor(iamService: IamService) {
this.#iamService = iamService; this.#iamService = iamService;
} }
public getRoles = async (req: Request, res: Response): Promise<void> => { public getRoles = async (req: Request, res: Response): Promise<void> => {
try { try {
const roles = await this.#iamService.getRoles(); const roles = await this.#iamService.getRoles();
res.status(200).json(roles); res.status(200).json(roles);
} catch (error) { } catch (error) {
res.status(500).json({ error: 'Failed to get roles.' }); res.status(500).json({ error: 'Failed to get roles.' });
} }
}; };
public getRoleById = async (req: Request, res: Response): Promise<void> => { public getRoleById = async (req: Request, res: Response): Promise<void> => {
const { id } = req.params; const { id } = req.params;
try { try {
const role = await this.#iamService.getRoleById(id); const role = await this.#iamService.getRoleById(id);
if (role) { if (role) {
res.status(200).json(role); res.status(200).json(role);
} else { } else {
res.status(404).json({ error: 'Role not found.' }); res.status(404).json({ error: 'Role not found.' });
} }
} catch (error) { } catch (error) {
res.status(500).json({ error: 'Failed to get role.' }); res.status(500).json({ error: 'Failed to get role.' });
} }
}; };
public createRole = async (req: Request, res: Response): Promise<void> => { public createRole = async (req: Request, res: Response): Promise<void> => {
const { name, policy } = req.body; const { name, policy } = req.body;
if (!name || !policy) { if (!name || !policy) {
res.status(400).json({ error: 'Missing required fields: name and policy.' }); res.status(400).json({ error: 'Missing required fields: name and policy.' });
return; return;
} }
for (const statement of policy) { for (const statement of policy) {
const { valid, reason } = PolicyValidator.isValid(statement as PolicyStatement); const { valid, reason } = PolicyValidator.isValid(statement as PolicyStatement);
if (!valid) { if (!valid) {
res.status(400).json({ error: `Invalid policy statement: ${reason}` }); res.status(400).json({ error: `Invalid policy statement: ${reason}` });
return; return;
} }
} }
try { try {
const role = await this.#iamService.createRole(name, policy); const role = await this.#iamService.createRole(name, policy);
res.status(201).json(role); res.status(201).json(role);
} catch (error) { } catch (error) {
res.status(500).json({ error: 'Failed to create role.' }); res.status(500).json({ error: 'Failed to create role.' });
} }
}; };
public deleteRole = async (req: Request, res: Response): Promise<void> => { public deleteRole = async (req: Request, res: Response): Promise<void> => {
const { id } = req.params; const { id } = req.params;
try { try {
await this.#iamService.deleteRole(id); await this.#iamService.deleteRole(id);
res.status(204).send(); res.status(204).send();
} catch (error) { } catch (error) {
res.status(500).json({ error: 'Failed to delete role.' }); res.status(500).json({ error: 'Failed to delete role.' });
} }
}; };
} }

278
packages/backend/src/api/controllers/ingestion.controller.ts
View File

@@ -1,153 +1,159 @@
import { Request, Response } from 'express'; import { Request, Response } from 'express';
import { IngestionService } from '../../services/IngestionService'; import { IngestionService } from '../../services/IngestionService';
import { import {
CreateIngestionSourceDto, CreateIngestionSourceDto,
UpdateIngestionSourceDto, UpdateIngestionSourceDto,
IngestionSource, IngestionSource,
SafeIngestionSource SafeIngestionSource,
} from '@open-archiver/types'; } from '@open-archiver/types';
import { logger } from '../../config/logger'; import { logger } from '../../config/logger';
import { config } from '../../config'; import { config } from '../../config';
export class IngestionController { export class IngestionController {
/** /**
* Converts an IngestionSource object to a safe version for client-side consumption * Converts an IngestionSource object to a safe version for client-side consumption
* by removing the credentials. * by removing the credentials.
* @param source The full IngestionSource object. * @param source The full IngestionSource object.
* @returns An object conforming to the SafeIngestionSource type. * @returns An object conforming to the SafeIngestionSource type.
*/ */
private toSafeIngestionSource(source: IngestionSource): SafeIngestionSource { private toSafeIngestionSource(source: IngestionSource): SafeIngestionSource {
const { credentials, ...safeSource } = source; const { credentials, ...safeSource } = source;
return safeSource; return safeSource;
} }
public create = async (req: Request, res: Response): Promise<Response> => { public create = async (req: Request, res: Response): Promise<Response> => {
if (config.app.isDemo) { if (config.app.isDemo) {
return res.status(403).json({ message: 'This operation is not allowed in demo mode.' }); return res.status(403).json({ message: 'This operation is not allowed in demo mode.' });
} }
try { try {
const dto: CreateIngestionSourceDto = req.body; const dto: CreateIngestionSourceDto = req.body;
const newSource = await IngestionService.create(dto); const newSource = await IngestionService.create(dto);
const safeSource = this.toSafeIngestionSource(newSource); const safeSource = this.toSafeIngestionSource(newSource);
return res.status(201).json(safeSource); return res.status(201).json(safeSource);
} catch (error: any) { } catch (error: any) {
logger.error({ err: error }, 'Create ingestion source error'); logger.error({ err: error }, 'Create ingestion source error');
// Return a 400 Bad Request for connection errors // Return a 400 Bad Request for connection errors
return res.status(400).json({ message: error.message || 'Failed to create ingestion source due to a connection error.' }); return res
} .status(400)
}; .json({
message:
error.message ||
'Failed to create ingestion source due to a connection error.',
});
}
};
public findAll = async (req: Request, res: Response): Promise<Response> => { public findAll = async (req: Request, res: Response): Promise<Response> => {
try { try {
const sources = await IngestionService.findAll(); const sources = await IngestionService.findAll();
const safeSources = sources.map(this.toSafeIngestionSource); const safeSources = sources.map(this.toSafeIngestionSource);
return res.status(200).json(safeSources); return res.status(200).json(safeSources);
} catch (error) { } catch (error) {
console.error('Find all ingestion sources error:', error); console.error('Find all ingestion sources error:', error);
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
public findById = async (req: Request, res: Response): Promise<Response> => { public findById = async (req: Request, res: Response): Promise<Response> => {
try { try {
const { id } = req.params; const { id } = req.params;
const source = await IngestionService.findById(id); const source = await IngestionService.findById(id);
const safeSource = this.toSafeIngestionSource(source); const safeSource = this.toSafeIngestionSource(source);
return res.status(200).json(safeSource); return res.status(200).json(safeSource);
} catch (error) { } catch (error) {
console.error(`Find ingestion source by id ${req.params.id} error:`, error); console.error(`Find ingestion source by id ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') { if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message }); return res.status(404).json({ message: error.message });
} }
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
public update = async (req: Request, res: Response): Promise<Response> => { public update = async (req: Request, res: Response): Promise<Response> => {
if (config.app.isDemo) { if (config.app.isDemo) {
return res.status(403).json({ message: 'This operation is not allowed in demo mode.' }); return res.status(403).json({ message: 'This operation is not allowed in demo mode.' });
} }
try { try {
const { id } = req.params; const { id } = req.params;
const dto: UpdateIngestionSourceDto = req.body; const dto: UpdateIngestionSourceDto = req.body;
const updatedSource = await IngestionService.update(id, dto); const updatedSource = await IngestionService.update(id, dto);
const safeSource = this.toSafeIngestionSource(updatedSource); const safeSource = this.toSafeIngestionSource(updatedSource);
return res.status(200).json(safeSource); return res.status(200).json(safeSource);
} catch (error) { } catch (error) {
console.error(`Update ingestion source ${req.params.id} error:`, error); console.error(`Update ingestion source ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') { if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message }); return res.status(404).json({ message: error.message });
} }
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
public delete = async (req: Request, res: Response): Promise<Response> => { public delete = async (req: Request, res: Response): Promise<Response> => {
if (config.app.isDemo) { if (config.app.isDemo) {
return res.status(403).json({ message: 'This operation is not allowed in demo mode.' }); return res.status(403).json({ message: 'This operation is not allowed in demo mode.' });
} }
try { try {
const { id } = req.params; const { id } = req.params;
await IngestionService.delete(id); await IngestionService.delete(id);
return res.status(204).send(); return res.status(204).send();
} catch (error) { } catch (error) {
console.error(`Delete ingestion source ${req.params.id} error:`, error); console.error(`Delete ingestion source ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') { if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message }); return res.status(404).json({ message: error.message });
} }
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
public triggerInitialImport = async (req: Request, res: Response): Promise<Response> => { public triggerInitialImport = async (req: Request, res: Response): Promise<Response> => {
if (config.app.isDemo) { if (config.app.isDemo) {
return res.status(403).json({ message: 'This operation is not allowed in demo mode.' }); return res.status(403).json({ message: 'This operation is not allowed in demo mode.' });
} }
try { try {
const { id } = req.params; const { id } = req.params;
await IngestionService.triggerInitialImport(id); await IngestionService.triggerInitialImport(id);
return res.status(202).json({ message: 'Initial import triggered successfully.' }); return res.status(202).json({ message: 'Initial import triggered successfully.' });
} catch (error) { } catch (error) {
console.error(`Trigger initial import for ${req.params.id} error:`, error); console.error(`Trigger initial import for ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') { if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message }); return res.status(404).json({ message: error.message });
} }
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
public pause = async (req: Request, res: Response): Promise<Response> => { public pause = async (req: Request, res: Response): Promise<Response> => {
if (config.app.isDemo) { if (config.app.isDemo) {
return res.status(403).json({ message: 'This operation is not allowed in demo mode.' }); return res.status(403).json({ message: 'This operation is not allowed in demo mode.' });
} }
try { try {
const { id } = req.params; const { id } = req.params;
const updatedSource = await IngestionService.update(id, { status: 'paused' }); const updatedSource = await IngestionService.update(id, { status: 'paused' });
const safeSource = this.toSafeIngestionSource(updatedSource); const safeSource = this.toSafeIngestionSource(updatedSource);
return res.status(200).json(safeSource); return res.status(200).json(safeSource);
} catch (error) { } catch (error) {
console.error(`Pause ingestion source ${req.params.id} error:`, error); console.error(`Pause ingestion source ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') { if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message }); return res.status(404).json({ message: error.message });
} }
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
public triggerForceSync = async (req: Request, res: Response): Promise<Response> => { public triggerForceSync = async (req: Request, res: Response): Promise<Response> => {
if (config.app.isDemo) { if (config.app.isDemo) {
return res.status(403).json({ message: 'This operation is not allowed in demo mode.' }); return res.status(403).json({ message: 'This operation is not allowed in demo mode.' });
} }
try { try {
const { id } = req.params; const { id } = req.params;
await IngestionService.triggerForceSync(id); await IngestionService.triggerForceSync(id);
return res.status(202).json({ message: 'Force sync triggered successfully.' }); return res.status(202).json({ message: 'Force sync triggered successfully.' });
} catch (error) { } catch (error) {
console.error(`Trigger force sync for ${req.params.id} error:`, error); console.error(`Trigger force sync for ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') { if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message }); return res.status(404).json({ message: error.message });
} }
return res.status(500).json({ message: 'An internal server error occurred' }); return res.status(500).json({ message: 'An internal server error occurred' });
} }
}; };
} }

46
packages/backend/src/api/controllers/search.controller.ts
View File

@@ -3,32 +3,32 @@ import { SearchService } from '../../services/SearchService';
import { MatchingStrategies } from 'meilisearch'; import { MatchingStrategies } from 'meilisearch';
export class SearchController { export class SearchController {
private searchService: SearchService; private searchService: SearchService;
constructor() { constructor() {
this.searchService = new SearchService(); this.searchService = new SearchService();
} }
public search = async (req: Request, res: Response): Promise<void> => { public search = async (req: Request, res: Response): Promise<void> => {
try { try {
const { keywords, page, limit, matchingStrategy } = req.query; const { keywords, page, limit, matchingStrategy } = req.query;
if (!keywords) { if (!keywords) {
res.status(400).json({ message: 'Keywords are required' }); res.status(400).json({ message: 'Keywords are required' });
return; return;
} }
const results = await this.searchService.searchEmails({ const results = await this.searchService.searchEmails({
query: keywords as string, query: keywords as string,
page: page ? parseInt(page as string) : 1, page: page ? parseInt(page as string) : 1,
limit: limit ? parseInt(limit as string) : 10, limit: limit ? parseInt(limit as string) : 10,
matchingStrategy: matchingStrategy as MatchingStrategies matchingStrategy: matchingStrategy as MatchingStrategies,
}); });
res.status(200).json(results); res.status(200).json(results);
} catch (error) { } catch (error) {
const message = error instanceof Error ? error.message : 'An unknown error occurred'; const message = error instanceof Error ? error.message : 'An unknown error occurred';
res.status(500).json({ message }); res.status(500).json({ message });
} }
}; };
} }

68
packages/backend/src/api/controllers/storage.controller.ts
View File

@@ -4,47 +4,47 @@ import * as path from 'path';
import { storage as storageConfig } from '../../config/storage'; import { storage as storageConfig } from '../../config/storage';
export class StorageController { export class StorageController {
constructor(private storageService: StorageService) { } constructor(private storageService: StorageService) {}
public downloadFile = async (req: Request, res: Response): Promise<void> => { public downloadFile = async (req: Request, res: Response): Promise<void> => {
const unsafePath = req.query.path as string; const unsafePath = req.query.path as string;
if (!unsafePath) { if (!unsafePath) {
res.status(400).send('File path is required'); res.status(400).send('File path is required');
return; return;
} }
// Normalize the path to prevent directory traversal // Normalize the path to prevent directory traversal
const normalizedPath = path.normalize(unsafePath).replace(/^(\.\.(\/|\\|$))+/, ''); const normalizedPath = path.normalize(unsafePath).replace(/^(\.\.(\/|\\|$))+/, '');
// Determine the base path from storage configuration // Determine the base path from storage configuration
const basePath = storageConfig.type === 'local' ? storageConfig.rootPath : '/'; const basePath = storageConfig.type === 'local' ? storageConfig.rootPath : '/';
// Resolve the full path and ensure it's within the storage directory // Resolve the full path and ensure it's within the storage directory
const fullPath = path.join(basePath, normalizedPath); const fullPath = path.join(basePath, normalizedPath);
if (!fullPath.startsWith(basePath)) { if (!fullPath.startsWith(basePath)) {
res.status(400).send('Invalid file path'); res.status(400).send('Invalid file path');
return; return;
} }
// Use the sanitized, relative path for storage service operations // Use the sanitized, relative path for storage service operations
const safePath = path.relative(basePath, fullPath); const safePath = path.relative(basePath, fullPath);
try { try {
const fileExists = await this.storageService.exists(safePath); const fileExists = await this.storageService.exists(safePath);
if (!fileExists) { if (!fileExists) {
res.status(404).send('File not found'); res.status(404).send('File not found');
return; return;
} }
const fileStream = await this.storageService.get(safePath); const fileStream = await this.storageService.get(safePath);
const fileName = path.basename(safePath); const fileName = path.basename(safePath);
res.setHeader('Content-Disposition', `attachment; filename="${fileName}"`); res.setHeader('Content-Disposition', `attachment; filename="${fileName}"`);
fileStream.pipe(res); fileStream.pipe(res);
} catch (error) { } catch (error) {
console.error('Error downloading file:', error); console.error('Error downloading file:', error);
res.status(500).send('Error downloading file'); res.status(500).send('Error downloading file');
} }
}; };
} }

29
packages/backend/src/api/controllers/upload.controller.ts
View File

@@ -4,23 +4,22 @@ import { randomUUID } from 'crypto';
import busboy from 'busboy'; import busboy from 'busboy';
import { config } from '../../config/index'; import { config } from '../../config/index';
export const uploadFile = async (req: Request, res: Response) => { export const uploadFile = async (req: Request, res: Response) => {
const storage = new StorageService(); const storage = new StorageService();
const bb = busboy({ headers: req.headers }); const bb = busboy({ headers: req.headers });
let filePath = ''; let filePath = '';
let originalFilename = ''; let originalFilename = '';
bb.on('file', (fieldname, file, filename) => { bb.on('file', (fieldname, file, filename) => {
originalFilename = filename.filename; originalFilename = filename.filename;
const uuid = randomUUID(); const uuid = randomUUID();
filePath = `${config.storage.openArchiverFolderName}/tmp/${uuid}-${originalFilename}`; filePath = `${config.storage.openArchiverFolderName}/tmp/${uuid}-${originalFilename}`;
storage.put(filePath, file); storage.put(filePath, file);
}); });
bb.on('finish', () => { bb.on('finish', () => {
res.json({ filePath }); res.json({ filePath });
}); });
req.pipe(bb); req.pipe(bb);
}; };

10
packages/backend/src/api/middleware/rateLimiter.ts
View File

@@ -2,9 +2,9 @@ import rateLimit from 'express-rate-limit';
// Rate limiter to prevent brute-force attacks on the login endpoint // Rate limiter to prevent brute-force attacks on the login endpoint
export const loginRateLimiter = rateLimit({ export const loginRateLimiter = rateLimit({
windowMs: 15 * 60 * 1000, // 15 minutes windowMs: 15 * 60 * 1000, // 15 minutes
max: 10, // Limit each IP to 10 login requests per windowMs max: 10, // Limit each IP to 10 login requests per windowMs
message: 'Too many login attempts from this IP, please try again after 15 minutes', message: 'Too many login attempts from this IP, please try again after 15 minutes',
standardHeaders: true, // Return rate limit info in the `RateLimit-*` headers standardHeaders: true, // Return rate limit info in the `RateLimit-*` headers
legacyHeaders: false, // Disable the `X-RateLimit-*` headers legacyHeaders: false, // Disable the `X-RateLimit-*` headers
}); });

58
packages/backend/src/api/middleware/requireAuth.ts
View File

@@ -5,35 +5,37 @@ import 'dotenv/config';
// By using module augmentation, we can add our custom 'user' property // By using module augmentation, we can add our custom 'user' property
// to the Express Request interface in a type-safe way. // to the Express Request interface in a type-safe way.
declare global { declare global {
namespace Express { namespace Express {
export interface Request { export interface Request {
user?: AuthTokenPayload; user?: AuthTokenPayload;
} }
} }
} }
export const requireAuth = (authService: AuthService) => { export const requireAuth = (authService: AuthService) => {
return async (req: Request, res: Response, next: NextFunction) => { return async (req: Request, res: Response, next: NextFunction) => {
const authHeader = req.headers.authorization; const authHeader = req.headers.authorization;
if (!authHeader || !authHeader.startsWith('Bearer ')) { if (!authHeader || !authHeader.startsWith('Bearer ')) {
return res.status(401).json({ message: 'Unauthorized: No token provided' }); return res.status(401).json({ message: 'Unauthorized: No token provided' });
} }
const token = authHeader.split(' ')[1]; const token = authHeader.split(' ')[1];
try { try {
// use a SUPER_API_KEY for all authentications. add process.env.SUPER_API_KEY conditional check in case user didn't set a SUPER_API_KEY. // use a SUPER_API_KEY for all authentications. add process.env.SUPER_API_KEY conditional check in case user didn't set a SUPER_API_KEY.
if (process.env.SUPER_API_KEY && token === process.env.SUPER_API_KEY) { if (process.env.SUPER_API_KEY && token === process.env.SUPER_API_KEY) {
next(); next();
return; return;
} }
const payload = await authService.verifyToken(token); const payload = await authService.verifyToken(token);
if (!payload) { if (!payload) {
return res.status(401).json({ message: 'Unauthorized: Invalid token' }); return res.status(401).json({ message: 'Unauthorized: Invalid token' });
} }
req.user = payload; req.user = payload;
next(); next();
} catch (error) { } catch (error) {
console.error('Authentication error:', error); console.error('Authentication error:', error);
return res.status(500).json({ message: 'An internal server error occurred during authentication' }); return res
} .status(500)
}; .json({ message: 'An internal server error occurred during authentication' });
}
};
}; };

18
packages/backend/src/api/routes/archived-email.routes.ts
View File

@@ -4,19 +4,19 @@ import { requireAuth } from '../middleware/requireAuth';
import { AuthService } from '../../services/AuthService'; import { AuthService } from '../../services/AuthService';
export const createArchivedEmailRouter = ( export const createArchivedEmailRouter = (
archivedEmailController: ArchivedEmailController, archivedEmailController: ArchivedEmailController,
authService: AuthService authService: AuthService
): Router => { ): Router => {
const router = Router(); const router = Router();
// Secure all routes in this module // Secure all routes in this module
router.use(requireAuth(authService)); router.use(requireAuth(authService));
router.get('/ingestion-source/:ingestionSourceId', archivedEmailController.getArchivedEmails); router.get('/ingestion-source/:ingestionSourceId', archivedEmailController.getArchivedEmails);
router.get('/:id', archivedEmailController.getArchivedEmailById); router.get('/:id', archivedEmailController.getArchivedEmailById);
router.delete('/:id', archivedEmailController.deleteArchivedEmail); router.delete('/:id', archivedEmailController.deleteArchivedEmail);
return router; return router;
}; };

40
packages/backend/src/api/routes/auth.routes.ts
View File

@@ -3,28 +3,28 @@ import { loginRateLimiter } from '../middleware/rateLimiter';
import type { AuthController } from '../controllers/auth.controller'; import type { AuthController } from '../controllers/auth.controller';
export const createAuthRouter = (authController: AuthController): Router => { export const createAuthRouter = (authController: AuthController): Router => {
const router = Router(); const router = Router();
/** /**
* @route POST /api/v1/auth/setup * @route POST /api/v1/auth/setup
* @description Creates the initial administrator user. * @description Creates the initial administrator user.
* @access Public * @access Public
*/ */
router.post('/setup', loginRateLimiter, authController.setup); router.post('/setup', loginRateLimiter, authController.setup);
/** /**
* @route POST /api/v1/auth/login * @route POST /api/v1/auth/login
* @description Authenticates a user and returns a JWT. * @description Authenticates a user and returns a JWT.
* @access Public * @access Public
*/ */
router.post('/login', loginRateLimiter, authController.login); router.post('/login', loginRateLimiter, authController.login);
/** /**
* @route GET /api/v1/auth/status * @route GET /api/v1/auth/status
* @description Checks if the application has been set up. * @description Checks if the application has been set up.
* @access Public * @access Public
*/ */
router.get('/status', authController.status); router.get('/status', authController.status);
return router; return router;
}; };

16
packages/backend/src/api/routes/dashboard.routes.ts
View File

@@ -4,15 +4,15 @@ import { requireAuth } from '../middleware/requireAuth';
import { AuthService } from '../../services/AuthService'; import { AuthService } from '../../services/AuthService';
export const createDashboardRouter = (authService: AuthService): Router => { export const createDashboardRouter = (authService: AuthService): Router => {
const router = Router(); const router = Router();
router.use(requireAuth(authService)); router.use(requireAuth(authService));
router.get('/stats', dashboardController.getStats); router.get('/stats', dashboardController.getStats);
router.get('/ingestion-history', dashboardController.getIngestionHistory); router.get('/ingestion-history', dashboardController.getIngestionHistory);
router.get('/ingestion-sources', dashboardController.getIngestionSources); router.get('/ingestion-sources', dashboardController.getIngestionSources);
router.get('/recent-syncs', dashboardController.getRecentSyncs); router.get('/recent-syncs', dashboardController.getRecentSyncs);
router.get('/indexed-insights', dashboardController.getIndexedInsights); router.get('/indexed-insights', dashboardController.getIndexedInsights);
return router; return router;
}; };

52
packages/backend/src/api/routes/iam.routes.ts
View File

@@ -3,34 +3,34 @@ import { requireAuth } from '../middleware/requireAuth';
import type { IamController } from '../controllers/iam.controller'; import type { IamController } from '../controllers/iam.controller';
export const createIamRouter = (iamController: IamController): Router => { export const createIamRouter = (iamController: IamController): Router => {
const router = Router(); const router = Router();
/** /**
* @route GET /api/v1/iam/roles * @route GET /api/v1/iam/roles
* @description Gets all roles. * @description Gets all roles.
* @access Private * @access Private
*/ */
router.get('/roles', requireAuth, iamController.getRoles); router.get('/roles', requireAuth, iamController.getRoles);
/** /**
* @route GET /api/v1/iam/roles/:id * @route GET /api/v1/iam/roles/:id
* @description Gets a role by ID. * @description Gets a role by ID.
* @access Private * @access Private
*/ */
router.get('/roles/:id', requireAuth, iamController.getRoleById); router.get('/roles/:id', requireAuth, iamController.getRoleById);
/** /**
* @route POST /api/v1/iam/roles * @route POST /api/v1/iam/roles
* @description Creates a new role. * @description Creates a new role.
* @access Private * @access Private
*/ */
router.post('/roles', requireAuth, iamController.createRole); router.post('/roles', requireAuth, iamController.createRole);
/** /**
* @route DELETE /api/v1/iam/roles/:id * @route DELETE /api/v1/iam/roles/:id
* @description Deletes a role. * @description Deletes a role.
* @access Private * @access Private
*/ */
router.delete('/roles/:id', requireAuth, iamController.deleteRole); router.delete('/roles/:id', requireAuth, iamController.deleteRole);
return router; return router;
}; };

28
packages/backend/src/api/routes/ingestion.routes.ts
View File

@@ -4,29 +4,29 @@ import { requireAuth } from '../middleware/requireAuth';
import { AuthService } from '../../services/AuthService'; import { AuthService } from '../../services/AuthService';
export const createIngestionRouter = ( export const createIngestionRouter = (
ingestionController: IngestionController, ingestionController: IngestionController,
authService: AuthService authService: AuthService
): Router => { ): Router => {
const router = Router(); const router = Router();
// Secure all routes in this module // Secure all routes in this module
router.use(requireAuth(authService)); router.use(requireAuth(authService));
router.post('/', ingestionController.create); router.post('/', ingestionController.create);
router.get('/', ingestionController.findAll); router.get('/', ingestionController.findAll);
router.get('/:id', ingestionController.findById); router.get('/:id', ingestionController.findById);
router.put('/:id', ingestionController.update); router.put('/:id', ingestionController.update);
router.delete('/:id', ingestionController.delete); router.delete('/:id', ingestionController.delete);
router.post('/:id/import', ingestionController.triggerInitialImport); router.post('/:id/import', ingestionController.triggerInitialImport);
router.post('/:id/pause', ingestionController.pause); router.post('/:id/pause', ingestionController.pause);
router.post('/:id/sync', ingestionController.triggerForceSync); router.post('/:id/sync', ingestionController.triggerForceSync);
return router; return router;
}; };

12
packages/backend/src/api/routes/search.routes.ts
View File

@@ -4,14 +4,14 @@ import { requireAuth } from '../middleware/requireAuth';
import { AuthService } from '../../services/AuthService'; import { AuthService } from '../../services/AuthService';
export const createSearchRouter = ( export const createSearchRouter = (
searchController: SearchController, searchController: SearchController,
authService: AuthService authService: AuthService
): Router => { ): Router => {
const router = Router(); const router = Router();
router.use(requireAuth(authService)); router.use(requireAuth(authService));
router.get('/', searchController.search); router.get('/', searchController.search);
return router; return router;
}; };

14
packages/backend/src/api/routes/storage.routes.ts
View File

@@ -4,15 +4,15 @@ import { requireAuth } from '../middleware/requireAuth';
import { AuthService } from '../../services/AuthService'; import { AuthService } from '../../services/AuthService';
export const createStorageRouter = ( export const createStorageRouter = (
storageController: StorageController, storageController: StorageController,
authService: AuthService authService: AuthService
): Router => { ): Router => {
const router = Router(); const router = Router();
// Secure all routes in this module // Secure all routes in this module
router.use(requireAuth(authService)); router.use(requireAuth(authService));
router.get('/download', storageController.downloadFile); router.get('/download', storageController.downloadFile);
return router; return router;
}; };

3
packages/backend/src/api/routes/test.routes.ts
View File

@@ -3,7 +3,4 @@ import { ingestionQueue } from '../../jobs/queues';
const router: Router = Router(); const router: Router = Router();
export default router; export default router;

8
packages/backend/src/api/routes/upload.routes.ts
View File

@@ -4,11 +4,11 @@ import { requireAuth } from '../middleware/requireAuth';
import { AuthService } from '../../services/AuthService'; import { AuthService } from '../../services/AuthService';
export const createUploadRouter = (authService: AuthService): Router => { export const createUploadRouter = (authService: AuthService): Router => {
const router = Router(); const router = Router();
router.use(requireAuth(authService)); router.use(requireAuth(authService));
router.post('/', uploadFile); router.post('/', uploadFile);
return router; return router;
}; };

10
packages/backend/src/config/app.ts
View File

@@ -1,9 +1,9 @@
import 'dotenv/config'; import 'dotenv/config';
export const app = { export const app = {
nodeEnv: process.env.NODE_ENV || 'development', nodeEnv: process.env.NODE_ENV || 'development',
port: process.env.PORT_BACKEND ? parseInt(process.env.PORT_BACKEND, 10) : 4000, port: process.env.PORT_BACKEND ? parseInt(process.env.PORT_BACKEND, 10) : 4000,
encryptionKey: process.env.ENCRYPTION_KEY, encryptionKey: process.env.ENCRYPTION_KEY,
isDemo: process.env.IS_DEMO === 'true', isDemo: process.env.IS_DEMO === 'true',
syncFrequency: process.env.SYNC_FREQUENCY || '* * * * *' //default to 1 minute syncFrequency: process.env.SYNC_FREQUENCY || '* * * * *', //default to 1 minute
}; };

8
packages/backend/src/config/index.ts
View File

@@ -4,8 +4,8 @@ import { searchConfig } from './search';
import { connection as redisConfig } from './redis'; import { connection as redisConfig } from './redis';
export const config = { export const config = {
storage, storage,
app, app,
search: searchConfig, search: searchConfig,
redis: redisConfig, redis: redisConfig,
}; };

14
packages/backend/src/config/logger.ts
View File

@@ -1,11 +1,11 @@
import pino from 'pino'; import pino from 'pino';
export const logger = pino({ export const logger = pino({
level: process.env.LOG_LEVEL || 'info', level: process.env.LOG_LEVEL || 'info',
transport: { transport: {
target: 'pino-pretty', target: 'pino-pretty',
options: { options: {
colorize: true colorize: true,
} },
} },
}); });

14
packages/backend/src/config/redis.ts
View File

@@ -4,16 +4,16 @@ import 'dotenv/config';
* @see https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/guide/connections.md * @see https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/guide/connections.md
*/ */
const connectionOptions: any = { const connectionOptions: any = {
host: process.env.REDIS_HOST || 'localhost', host: process.env.REDIS_HOST || 'localhost',
port: (process.env.REDIS_PORT && parseInt(process.env.REDIS_PORT, 10)) || 6379, port: (process.env.REDIS_PORT && parseInt(process.env.REDIS_PORT, 10)) || 6379,
password: process.env.REDIS_PASSWORD, password: process.env.REDIS_PASSWORD,
enableReadyCheck: true, enableReadyCheck: true,
}; };
if (process.env.REDIS_TLS_ENABLED === 'true') { if (process.env.REDIS_TLS_ENABLED === 'true') {
connectionOptions.tls = { connectionOptions.tls = {
rejectUnauthorized: false rejectUnauthorized: false,
}; };
} }
export const connection = connectionOptions; export const connection = connectionOptions;

4
packages/backend/src/config/search.ts
View File

@@ -1,6 +1,6 @@
import 'dotenv/config'; import 'dotenv/config';
export const searchConfig = { export const searchConfig = {
host: process.env.MEILI_HOST || 'http://127.0.0.1:7700', host: process.env.MEILI_HOST || 'http://127.0.0.1:7700',
apiKey: process.env.MEILI_MASTER_KEY || '', apiKey: process.env.MEILI_MASTER_KEY || '',
}; };

54
packages/backend/src/config/storage.ts
View File

@@ -6,35 +6,35 @@ const openArchiverFolderName = 'open-archiver';
let storageConfig: StorageConfig; let storageConfig: StorageConfig;
if (storageType === 'local') { if (storageType === 'local') {
if (!process.env.STORAGE_LOCAL_ROOT_PATH) { if (!process.env.STORAGE_LOCAL_ROOT_PATH) {
throw new Error('STORAGE_LOCAL_ROOT_PATH is not defined in the environment variables'); throw new Error('STORAGE_LOCAL_ROOT_PATH is not defined in the environment variables');
} }
storageConfig = { storageConfig = {
type: 'local', type: 'local',
rootPath: process.env.STORAGE_LOCAL_ROOT_PATH, rootPath: process.env.STORAGE_LOCAL_ROOT_PATH,
openArchiverFolderName: openArchiverFolderName openArchiverFolderName: openArchiverFolderName,
}; };
} else if (storageType === 's3') { } else if (storageType === 's3') {
if ( if (
!process.env.STORAGE_S3_ENDPOINT || !process.env.STORAGE_S3_ENDPOINT ||
!process.env.STORAGE_S3_BUCKET || !process.env.STORAGE_S3_BUCKET ||
!process.env.STORAGE_S3_ACCESS_KEY_ID || !process.env.STORAGE_S3_ACCESS_KEY_ID ||
!process.env.STORAGE_S3_SECRET_ACCESS_KEY !process.env.STORAGE_S3_SECRET_ACCESS_KEY
) { ) {
throw new Error('One or more S3 storage environment variables are not defined'); throw new Error('One or more S3 storage environment variables are not defined');
} }
storageConfig = { storageConfig = {
type: 's3', type: 's3',
endpoint: process.env.STORAGE_S3_ENDPOINT, endpoint: process.env.STORAGE_S3_ENDPOINT,
bucket: process.env.STORAGE_S3_BUCKET, bucket: process.env.STORAGE_S3_BUCKET,
accessKeyId: process.env.STORAGE_S3_ACCESS_KEY_ID, accessKeyId: process.env.STORAGE_S3_ACCESS_KEY_ID,
secretAccessKey: process.env.STORAGE_S3_SECRET_ACCESS_KEY, secretAccessKey: process.env.STORAGE_S3_SECRET_ACCESS_KEY,
region: process.env.STORAGE_S3_REGION, region: process.env.STORAGE_S3_REGION,
forcePathStyle: process.env.STORAGE_S3_FORCE_PATH_STYLE === 'true', forcePathStyle: process.env.STORAGE_S3_FORCE_PATH_STYLE === 'true',
openArchiverFolderName: openArchiverFolderName openArchiverFolderName: openArchiverFolderName,
}; };
} else { } else {
throw new Error(`Invalid STORAGE_TYPE: ${storageType}`); throw new Error(`Invalid STORAGE_TYPE: ${storageType}`);
} }
export const storage = storageConfig; export const storage = storageConfig;

2
packages/backend/src/database/index.ts
View File

@@ -6,7 +6,7 @@ import * as schema from './schema';
import { encodeDatabaseUrl } from '../helpers/db'; import { encodeDatabaseUrl } from '../helpers/db';
if (!process.env.DATABASE_URL) { if (!process.env.DATABASE_URL) {
throw new Error('DATABASE_URL is not set in the .env file'); throw new Error('DATABASE_URL is not set in the .env file');
} }
const connectionString = encodeDatabaseUrl(process.env.DATABASE_URL); const connectionString = encodeDatabaseUrl(process.env.DATABASE_URL);

24
packages/backend/src/database/migrate.ts
View File

@@ -7,23 +7,23 @@ import { encodeDatabaseUrl } from '../helpers/db';
config(); config();
const runMigrate = async () => { const runMigrate = async () => {
if (!process.env.DATABASE_URL) { if (!process.env.DATABASE_URL) {
throw new Error('DATABASE_URL is not set in the .env file'); throw new Error('DATABASE_URL is not set in the .env file');
} }
const connectionString = encodeDatabaseUrl(process.env.DATABASE_URL); const connectionString = encodeDatabaseUrl(process.env.DATABASE_URL);
const connection = postgres(connectionString, { max: 1 }); const connection = postgres(connectionString, { max: 1 });
const db = drizzle(connection); const db = drizzle(connection);
console.log('Running migrations...'); console.log('Running migrations...');
await migrate(db, { migrationsFolder: 'src/database/migrations' }); await migrate(db, { migrationsFolder: 'src/database/migrations' });
console.log('Migrations completed!'); console.log('Migrations completed!');
process.exit(0); process.exit(0);
}; };
runMigrate().catch((err) => { runMigrate().catch((err) => {
console.error('Migration failed!', err); console.error('Migration failed!', err);
process.exit(1); process.exit(1);
}); });

1586
packages/backend/src/database/migrations/meta/0000_snapshot.json
View File

File diff suppressed because it is too large Load Diff

1587
packages/backend/src/database/migrations/meta/0001_snapshot.json
View File

File diff suppressed because it is too large Load Diff

1587
packages/backend/src/database/migrations/meta/0002_snapshot.json
View File

File diff suppressed because it is too large Load Diff

1587
packages/backend/src/database/migrations/meta/0003_snapshot.json
View File

File diff suppressed because it is too large Load Diff

1596
packages/backend/src/database/migrations/meta/0004_snapshot.json
View File

File diff suppressed because it is too large Load Diff

1596
packages/backend/src/database/migrations/meta/0005_snapshot.json
View File

File diff suppressed because it is too large Load Diff

1608
packages/backend/src/database/migrations/meta/0006_snapshot.json
View File

File diff suppressed because it is too large Load Diff

1608
packages/backend/src/database/migrations/meta/0007_snapshot.json
View File

File diff suppressed because it is too large Load Diff

1664
packages/backend/src/database/migrations/meta/0009_snapshot.json
View File

File diff suppressed because it is too large Load Diff

2111
packages/backend/src/database/migrations/meta/0010_snapshot.json
View File

File diff suppressed because it is too large Load Diff

2123
packages/backend/src/database/migrations/meta/0011_snapshot.json
View File

File diff suppressed because it is too large Load Diff

2126
packages/backend/src/database/migrations/meta/0012_snapshot.json
View File

File diff suppressed because it is too large Load Diff

2150
packages/backend/src/database/migrations/meta/0013_snapshot.json
View File

File diff suppressed because it is too large Load Diff

2157
packages/backend/src/database/migrations/meta/0014_snapshot.json
View File

File diff suppressed because it is too large Load Diff

220
packages/backend/src/database/migrations/meta/_journal.json
View File

@@ -1,111 +1,111 @@
{ {
"version": "7", "version": "7",
"dialect": "postgresql", "dialect": "postgresql",
"entries": [ "entries": [
{ {
"idx": 0, "idx": 0,
"version": "7", "version": "7",
"when": 1752225352591, "when": 1752225352591,
"tag": "0000_amusing_namora", "tag": "0000_amusing_namora",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 1, "idx": 1,
"version": "7", "version": "7",
"when": 1752326803882, "when": 1752326803882,
"tag": "0001_odd_night_thrasher", "tag": "0001_odd_night_thrasher",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 2, "idx": 2,
"version": "7", "version": "7",
"when": 1752332648392, "when": 1752332648392,
"tag": "0002_lethal_quentin_quire", "tag": "0002_lethal_quentin_quire",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 3, "idx": 3,
"version": "7", "version": "7",
"when": 1752332967084, "when": 1752332967084,
"tag": "0003_petite_wrecker", "tag": "0003_petite_wrecker",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 4, "idx": 4,
"version": "7", "version": "7",
"when": 1752606108876, "when": 1752606108876,
"tag": "0004_sleepy_paper_doll", "tag": "0004_sleepy_paper_doll",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 5, "idx": 5,
"version": "7", "version": "7",
"when": 1752606327253, "when": 1752606327253,
"tag": "0005_chunky_sue_storm", "tag": "0005_chunky_sue_storm",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 6, "idx": 6,
"version": "7", "version": "7",
"when": 1753112018514, "when": 1753112018514,
"tag": "0006_majestic_caretaker", "tag": "0006_majestic_caretaker",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 7, "idx": 7,
"version": "7", "version": "7",
"when": 1753190159356, "when": 1753190159356,
"tag": "0007_handy_archangel", "tag": "0007_handy_archangel",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 8, "idx": 8,
"version": "7", "version": "7",
"when": 1753370737317, "when": 1753370737317,
"tag": "0008_eminent_the_spike", "tag": "0008_eminent_the_spike",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 9, "idx": 9,
"version": "7", "version": "7",
"when": 1754337938241, "when": 1754337938241,
"tag": "0009_late_lenny_balinger", "tag": "0009_late_lenny_balinger",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 10, "idx": 10,
"version": "7", "version": "7",
"when": 1754420780849, "when": 1754420780849,
"tag": "0010_perpetual_lightspeed", "tag": "0010_perpetual_lightspeed",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 11, "idx": 11,
"version": "7", "version": "7",
"when": 1754422064158, "when": 1754422064158,
"tag": "0011_tan_blackheart", "tag": "0011_tan_blackheart",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 12, "idx": 12,
"version": "7", "version": "7",
"when": 1754476962901, "when": 1754476962901,
"tag": "0012_warm_the_stranger", "tag": "0012_warm_the_stranger",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 13, "idx": 13,
"version": "7", "version": "7",
"when": 1754659373517, "when": 1754659373517,
"tag": "0013_classy_talkback", "tag": "0013_classy_talkback",
"breakpoints": true "breakpoints": true
}, },
{ {
"idx": 14, "idx": 14,
"version": "7", "version": "7",
"when": 1754831765718, "when": 1754831765718,
"tag": "0014_foamy_vapor", "tag": "0014_foamy_vapor",
"breakpoints": true "breakpoints": true
} }
] ]
} }

58
packages/backend/src/database/schema/archived-emails.ts
View File

@@ -3,36 +3,36 @@ import { boolean, jsonb, pgTable, text, timestamp, uuid, bigint, index } from 'd
import { ingestionSources } from './ingestion-sources'; import { ingestionSources } from './ingestion-sources';
export const archivedEmails = pgTable( export const archivedEmails = pgTable(
'archived_emails', 'archived_emails',
{ {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
threadId: text('thread_id'), threadId: text('thread_id'),
ingestionSourceId: uuid('ingestion_source_id') ingestionSourceId: uuid('ingestion_source_id')
.notNull() .notNull()
.references(() => ingestionSources.id, { onDelete: 'cascade' }), .references(() => ingestionSources.id, { onDelete: 'cascade' }),
userEmail: text('user_email').notNull(), userEmail: text('user_email').notNull(),
messageIdHeader: text('message_id_header'), messageIdHeader: text('message_id_header'),
sentAt: timestamp('sent_at', { withTimezone: true }).notNull(), sentAt: timestamp('sent_at', { withTimezone: true }).notNull(),
subject: text('subject'), subject: text('subject'),
senderName: text('sender_name'), senderName: text('sender_name'),
senderEmail: text('sender_email').notNull(), senderEmail: text('sender_email').notNull(),
recipients: jsonb('recipients'), recipients: jsonb('recipients'),
storagePath: text('storage_path').notNull(), storagePath: text('storage_path').notNull(),
storageHashSha256: text('storage_hash_sha256').notNull(), storageHashSha256: text('storage_hash_sha256').notNull(),
sizeBytes: bigint('size_bytes', { mode: 'number' }).notNull(), sizeBytes: bigint('size_bytes', { mode: 'number' }).notNull(),
isIndexed: boolean('is_indexed').notNull().default(false), isIndexed: boolean('is_indexed').notNull().default(false),
hasAttachments: boolean('has_attachments').notNull().default(false), hasAttachments: boolean('has_attachments').notNull().default(false),
isOnLegalHold: boolean('is_on_legal_hold').notNull().default(false), isOnLegalHold: boolean('is_on_legal_hold').notNull().default(false),
archivedAt: timestamp('archived_at', { withTimezone: true }).notNull().defaultNow(), archivedAt: timestamp('archived_at', { withTimezone: true }).notNull().defaultNow(),
path: text('path'), path: text('path'),
tags: jsonb('tags'), tags: jsonb('tags'),
}, },
(table) => [index('thread_id_idx').on(table.threadId)] (table) => [index('thread_id_idx').on(table.threadId)]
); );
export const archivedEmailsRelations = relations(archivedEmails, ({ one }) => ({ export const archivedEmailsRelations = relations(archivedEmails, ({ one }) => ({
ingestionSource: one(ingestionSources, { ingestionSource: one(ingestionSources, {
fields: [archivedEmails.ingestionSourceId], fields: [archivedEmails.ingestionSourceId],
references: [ingestionSources.id] references: [ingestionSources.id],
}) }),
})); }));

50
packages/backend/src/database/schema/attachments.ts
View File

@@ -3,32 +3,40 @@ import { pgTable, text, uuid, bigint, primaryKey } from 'drizzle-orm/pg-core';
import { archivedEmails } from './archived-emails'; import { archivedEmails } from './archived-emails';
export const attachments = pgTable('attachments', { export const attachments = pgTable('attachments', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
filename: text('filename').notNull(), filename: text('filename').notNull(),
mimeType: text('mime_type'), mimeType: text('mime_type'),
sizeBytes: bigint('size_bytes', { mode: 'number' }).notNull(), sizeBytes: bigint('size_bytes', { mode: 'number' }).notNull(),
contentHashSha256: text('content_hash_sha256').notNull().unique(), contentHashSha256: text('content_hash_sha256').notNull().unique(),
storagePath: text('storage_path').notNull(), storagePath: text('storage_path').notNull(),
}); });
export const emailAttachments = pgTable('email_attachments', { export const emailAttachments = pgTable(
emailId: uuid('email_id').notNull().references(() => archivedEmails.id, { onDelete: 'cascade' }), 'email_attachments',
attachmentId: uuid('attachment_id').notNull().references(() => attachments.id, { onDelete: 'restrict' }), {
}, (t) => ({ emailId: uuid('email_id')
pk: primaryKey({ columns: [t.emailId, t.attachmentId] }), .notNull()
})); .references(() => archivedEmails.id, { onDelete: 'cascade' }),
attachmentId: uuid('attachment_id')
.notNull()
.references(() => attachments.id, { onDelete: 'restrict' }),
},
(t) => ({
pk: primaryKey({ columns: [t.emailId, t.attachmentId] }),
})
);
export const attachmentsRelations = relations(attachments, ({ many }) => ({ export const attachmentsRelations = relations(attachments, ({ many }) => ({
emailAttachments: many(emailAttachments), emailAttachments: many(emailAttachments),
})); }));
export const emailAttachmentsRelations = relations(emailAttachments, ({ one }) => ({ export const emailAttachmentsRelations = relations(emailAttachments, ({ one }) => ({
archivedEmail: one(archivedEmails, { archivedEmail: one(archivedEmails, {
fields: [emailAttachments.emailId], fields: [emailAttachments.emailId],
references: [archivedEmails.id], references: [archivedEmails.id],
}), }),
attachment: one(attachments, { attachment: one(attachments, {
fields: [emailAttachments.attachmentId], fields: [emailAttachments.attachmentId],
references: [attachments.id], references: [attachments.id],
}), }),
})); }));

16
packages/backend/src/database/schema/audit-logs.ts
View File

@@ -1,12 +1,12 @@
import { bigserial, boolean, jsonb, pgTable, text, timestamp } from 'drizzle-orm/pg-core'; import { bigserial, boolean, jsonb, pgTable, text, timestamp } from 'drizzle-orm/pg-core';
export const auditLogs = pgTable('audit_logs', { export const auditLogs = pgTable('audit_logs', {
id: bigserial('id', { mode: 'number' }).primaryKey(), id: bigserial('id', { mode: 'number' }).primaryKey(),
timestamp: timestamp('timestamp', { withTimezone: true }).notNull().defaultNow(), timestamp: timestamp('timestamp', { withTimezone: true }).notNull().defaultNow(),
actorIdentifier: text('actor_identifier').notNull(), actorIdentifier: text('actor_identifier').notNull(),
action: text('action').notNull(), action: text('action').notNull(),
targetType: text('target_type'), targetType: text('target_type'),
targetId: text('target_id'), targetId: text('target_id'),
details: jsonb('details'), details: jsonb('details'),
isTamperEvident: boolean('is_tamper_evident').default(false), isTamperEvident: boolean('is_tamper_evident').default(false),
}); });

114
packages/backend/src/database/schema/compliance.ts
View File

@@ -1,80 +1,94 @@
import { relations } from 'drizzle-orm'; import { relations } from 'drizzle-orm';
import { boolean, integer, jsonb, pgEnum, pgTable, text, timestamp, uuid } from 'drizzle-orm/pg-core'; import {
boolean,
integer,
jsonb,
pgEnum,
pgTable,
text,
timestamp,
uuid,
} from 'drizzle-orm/pg-core';
import { custodians } from './custodians'; import { custodians } from './custodians';
// --- Enums --- // --- Enums ---
export const retentionActionEnum = pgEnum('retention_action', ['delete_permanently', 'notify_admin']); export const retentionActionEnum = pgEnum('retention_action', [
'delete_permanently',
'notify_admin',
]);
// --- Tables --- // --- Tables ---
export const retentionPolicies = pgTable('retention_policies', { export const retentionPolicies = pgTable('retention_policies', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
name: text('name').notNull().unique(), name: text('name').notNull().unique(),
description: text('description'), description: text('description'),
priority: integer('priority').notNull(), priority: integer('priority').notNull(),
retentionPeriodDays: integer('retention_period_days').notNull(), retentionPeriodDays: integer('retention_period_days').notNull(),
actionOnExpiry: retentionActionEnum('action_on_expiry').notNull(), actionOnExpiry: retentionActionEnum('action_on_expiry').notNull(),
isEnabled: boolean('is_enabled').notNull().default(true), isEnabled: boolean('is_enabled').notNull().default(true),
conditions: jsonb('conditions'), conditions: jsonb('conditions'),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(), createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(), updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
}); });
export const ediscoveryCases = pgTable('ediscovery_cases', { export const ediscoveryCases = pgTable('ediscovery_cases', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
name: text('name').notNull().unique(), name: text('name').notNull().unique(),
description: text('description'), description: text('description'),
status: text('status').notNull().default('open'), status: text('status').notNull().default('open'),
createdByIdentifier: text('created_by_identifier').notNull(), createdByIdentifier: text('created_by_identifier').notNull(),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(), createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(), updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
}); });
export const legalHolds = pgTable('legal_holds', { export const legalHolds = pgTable('legal_holds', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
caseId: uuid('case_id').notNull().references(() => ediscoveryCases.id, { onDelete: 'cascade' }), caseId: uuid('case_id')
custodianId: uuid('custodian_id').references(() => custodians.id, { onDelete: 'cascade' }), .notNull()
holdCriteria: jsonb('hold_criteria'), .references(() => ediscoveryCases.id, { onDelete: 'cascade' }),
reason: text('reason'), custodianId: uuid('custodian_id').references(() => custodians.id, { onDelete: 'cascade' }),
appliedByIdentifier: text('applied_by_identifier').notNull(), holdCriteria: jsonb('hold_criteria'),
appliedAt: timestamp('applied_at', { withTimezone: true }).notNull().defaultNow(), reason: text('reason'),
removedAt: timestamp('removed_at', { withTimezone: true }), appliedByIdentifier: text('applied_by_identifier').notNull(),
appliedAt: timestamp('applied_at', { withTimezone: true }).notNull().defaultNow(),
removedAt: timestamp('removed_at', { withTimezone: true }),
}); });
export const exportJobs = pgTable('export_jobs', { export const exportJobs = pgTable('export_jobs', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
caseId: uuid('case_id').references(() => ediscoveryCases.id, { onDelete: 'set null' }), caseId: uuid('case_id').references(() => ediscoveryCases.id, { onDelete: 'set null' }),
format: text('format').notNull(), format: text('format').notNull(),
status: text('status').notNull().default('pending'), status: text('status').notNull().default('pending'),
query: jsonb('query').notNull(), query: jsonb('query').notNull(),
filePath: text('file_path'), filePath: text('file_path'),
createdByIdentifier: text('created_by_identifier').notNull(), createdByIdentifier: text('created_by_identifier').notNull(),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(), createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
completedAt: timestamp('completed_at', { withTimezone: true }), completedAt: timestamp('completed_at', { withTimezone: true }),
}); });
// --- Relations --- // --- Relations ---
export const ediscoveryCasesRelations = relations(ediscoveryCases, ({ many }) => ({ export const ediscoveryCasesRelations = relations(ediscoveryCases, ({ many }) => ({
legalHolds: many(legalHolds), legalHolds: many(legalHolds),
exportJobs: many(exportJobs), exportJobs: many(exportJobs),
})); }));
export const legalHoldsRelations = relations(legalHolds, ({ one }) => ({ export const legalHoldsRelations = relations(legalHolds, ({ one }) => ({
ediscoveryCase: one(ediscoveryCases, { ediscoveryCase: one(ediscoveryCases, {
fields: [legalHolds.caseId], fields: [legalHolds.caseId],
references: [ediscoveryCases.id], references: [ediscoveryCases.id],
}), }),
custodian: one(custodians, { custodian: one(custodians, {
fields: [legalHolds.custodianId], fields: [legalHolds.custodianId],
references: [custodians.id], references: [custodians.id],
}), }),
})); }));
export const exportJobsRelations = relations(exportJobs, ({ one }) => ({ export const exportJobsRelations = relations(exportJobs, ({ one }) => ({
ediscoveryCase: one(ediscoveryCases, { ediscoveryCase: one(ediscoveryCases, {
fields: [exportJobs.caseId], fields: [exportJobs.caseId],
references: [ediscoveryCases.id], references: [ediscoveryCases.id],
}), }),
})); }));

12
packages/backend/src/database/schema/custodians.ts
View File

@@ -2,10 +2,10 @@ import { pgTable, text, timestamp, uuid } from 'drizzle-orm/pg-core';
import { ingestionProviderEnum } from './ingestion-sources'; import { ingestionProviderEnum } from './ingestion-sources';
export const custodians = pgTable('custodians', { export const custodians = pgTable('custodians', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
email: text('email').notNull().unique(), email: text('email').notNull().unique(),
displayName: text('display_name'), displayName: text('display_name'),
sourceType: ingestionProviderEnum('source_type').notNull(), sourceType: ingestionProviderEnum('source_type').notNull(),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(), createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow() updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
}); });

48
packages/backend/src/database/schema/ingestion-sources.ts
View File

@@ -1,34 +1,34 @@
import { jsonb, pgEnum, pgTable, text, timestamp, uuid } from 'drizzle-orm/pg-core'; import { jsonb, pgEnum, pgTable, text, timestamp, uuid } from 'drizzle-orm/pg-core';
export const ingestionProviderEnum = pgEnum('ingestion_provider', [ export const ingestionProviderEnum = pgEnum('ingestion_provider', [
'google_workspace', 'google_workspace',
'microsoft_365', 'microsoft_365',
'generic_imap', 'generic_imap',
'pst_import', 'pst_import',
'eml_import' 'eml_import',
]); ]);
export const ingestionStatusEnum = pgEnum('ingestion_status', [ export const ingestionStatusEnum = pgEnum('ingestion_status', [
'active', 'active',
'paused', 'paused',
'error', 'error',
'pending_auth', 'pending_auth',
'syncing', 'syncing',
'importing', 'importing',
'auth_success', 'auth_success',
'imported' 'imported',
]); ]);
export const ingestionSources = pgTable('ingestion_sources', { export const ingestionSources = pgTable('ingestion_sources', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
name: text('name').notNull(), name: text('name').notNull(),
provider: ingestionProviderEnum('provider').notNull(), provider: ingestionProviderEnum('provider').notNull(),
credentials: text('credentials'), credentials: text('credentials'),
status: ingestionStatusEnum('status').notNull().default('pending_auth'), status: ingestionStatusEnum('status').notNull().default('pending_auth'),
lastSyncStartedAt: timestamp('last_sync_started_at', { withTimezone: true }), lastSyncStartedAt: timestamp('last_sync_started_at', { withTimezone: true }),
lastSyncFinishedAt: timestamp('last_sync_finished_at', { withTimezone: true }), lastSyncFinishedAt: timestamp('last_sync_finished_at', { withTimezone: true }),
lastSyncStatusMessage: text('last_sync_status_message'), lastSyncStatusMessage: text('last_sync_status_message'),
syncState: jsonb('sync_state'), syncState: jsonb('sync_state'),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(), createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow() updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
}); });

96
packages/backend/src/database/schema/users.ts
View File

@@ -1,27 +1,20 @@
import { relations, sql } from 'drizzle-orm'; import { relations, sql } from 'drizzle-orm';
import { import { pgTable, text, timestamp, uuid, primaryKey, jsonb } from 'drizzle-orm/pg-core';
pgTable,
text,
timestamp,
uuid,
primaryKey,
jsonb
} from 'drizzle-orm/pg-core';
import type { PolicyStatement } from '@open-archiver/types'; import type { PolicyStatement } from '@open-archiver/types';
/** /**
* The `users` table stores the core user information for authentication and identification. * The `users` table stores the core user information for authentication and identification.
*/ */
export const users = pgTable('users', { export const users = pgTable('users', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
email: text('email').notNull().unique(), email: text('email').notNull().unique(),
first_name: text('first_name'), first_name: text('first_name'),
last_name: text('last_name'), last_name: text('last_name'),
password: text('password'), password: text('password'),
provider: text('provider').default('local'), provider: text('provider').default('local'),
providerId: text('provider_id'), providerId: text('provider_id'),
createdAt: timestamp('created_at').defaultNow().notNull(), createdAt: timestamp('created_at').defaultNow().notNull(),
updatedAt: timestamp('updated_at').defaultNow().notNull() updatedAt: timestamp('updated_at').defaultNow().notNull(),
}); });
/** /**
@@ -29,14 +22,14 @@ export const users = pgTable('users', {
* It links a session to a user and records its expiration time. * It links a session to a user and records its expiration time.
*/ */
export const sessions = pgTable('sessions', { export const sessions = pgTable('sessions', {
id: text('id').primaryKey(), id: text('id').primaryKey(),
userId: uuid('user_id') userId: uuid('user_id')
.notNull() .notNull()
.references(() => users.id, { onDelete: 'cascade' }), .references(() => users.id, { onDelete: 'cascade' }),
expiresAt: timestamp('expires_at', { expiresAt: timestamp('expires_at', {
withTimezone: true, withTimezone: true,
mode: 'date' mode: 'date',
}).notNull() }).notNull(),
}); });
/** /**
@@ -44,11 +37,14 @@ export const sessions = pgTable('sessions', {
* Each role has a name and a set of policies that define its permissions. * Each role has a name and a set of policies that define its permissions.
*/ */
export const roles = pgTable('roles', { export const roles = pgTable('roles', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
name: text('name').notNull().unique(), name: text('name').notNull().unique(),
policies: jsonb('policies').$type<PolicyStatement[]>().notNull().default(sql`'[]'::jsonb`), policies: jsonb('policies')
createdAt: timestamp('created_at').defaultNow().notNull(), .$type<PolicyStatement[]>()
updatedAt: timestamp('updated_at').defaultNow().notNull() .notNull()
.default(sql`'[]'::jsonb`),
createdAt: timestamp('created_at').defaultNow().notNull(),
updatedAt: timestamp('updated_at').defaultNow().notNull(),
}); });
/** /**
@@ -56,34 +52,34 @@ export const roles = pgTable('roles', {
* This many-to-many relationship allows a user to have multiple roles. * This many-to-many relationship allows a user to have multiple roles.
*/ */
export const userRoles = pgTable( export const userRoles = pgTable(
'user_roles', 'user_roles',
{ {
userId: uuid('user_id') userId: uuid('user_id')
.notNull() .notNull()
.references(() => users.id, { onDelete: 'cascade' }), .references(() => users.id, { onDelete: 'cascade' }),
roleId: uuid('role_id') roleId: uuid('role_id')
.notNull() .notNull()
.references(() => roles.id, { onDelete: 'cascade' }) .references(() => roles.id, { onDelete: 'cascade' }),
}, },
(t) => [primaryKey({ columns: [t.userId, t.roleId] })] (t) => [primaryKey({ columns: [t.userId, t.roleId] })]
); );
// Define relationships for Drizzle ORM // Define relationships for Drizzle ORM
export const usersRelations = relations(users, ({ many }) => ({ export const usersRelations = relations(users, ({ many }) => ({
userRoles: many(userRoles) userRoles: many(userRoles),
})); }));
export const rolesRelations = relations(roles, ({ many }) => ({ export const rolesRelations = relations(roles, ({ many }) => ({
userRoles: many(userRoles) userRoles: many(userRoles),
})); }));
export const userRolesRelations = relations(userRoles, ({ one }) => ({ export const userRolesRelations = relations(userRoles, ({ one }) => ({
role: one(roles, { role: one(roles, {
fields: [userRoles.roleId], fields: [userRoles.roleId],
references: [roles.id] references: [roles.id],
}), }),
user: one(users, { user: one(users, {
fields: [userRoles.userId], fields: [userRoles.userId],
references: [users.id] references: [users.id],
}) }),
})); }));

20
packages/backend/src/helpers/db.ts
View File

@@ -1,12 +1,12 @@
export const encodeDatabaseUrl = (databaseUrl: string): string => { export const encodeDatabaseUrl = (databaseUrl: string): string => {
try { try {
const url = new URL(databaseUrl); const url = new URL(databaseUrl);
if (url.password) { if (url.password) {
url.password = encodeURIComponent(url.password); url.password = encodeURIComponent(url.password);
} }
return url.toString(); return url.toString();
} catch (error) { } catch (error) {
console.error("Invalid DATABASE_URL, please check your .env file.", error); console.error('Invalid DATABASE_URL, please check your .env file.', error);
throw new Error("Invalid DATABASE_URL"); throw new Error('Invalid DATABASE_URL');
} }
}; };

12
packages/backend/src/helpers/streamToBuffer.ts
View File

@@ -1,8 +1,8 @@
export function streamToBuffer(stream: NodeJS.ReadableStream): Promise<Buffer> { export function streamToBuffer(stream: NodeJS.ReadableStream): Promise<Buffer> {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
const chunks: Buffer[] = []; const chunks: Buffer[] = [];
stream.on('data', (chunk) => chunks.push(chunk)); stream.on('data', (chunk) => chunks.push(chunk));
stream.on('error', reject); stream.on('error', reject);
stream.on('end', () => resolve(Buffer.concat(chunks))); stream.on('end', () => resolve(Buffer.concat(chunks)));
}); });
} }

120
packages/backend/src/helpers/textExtractor.ts
View File

@@ -3,80 +3,68 @@ import mammoth from 'mammoth';
import xlsx from 'xlsx'; import xlsx from 'xlsx';
function extractTextFromPdf(buffer: Buffer): Promise<string> { function extractTextFromPdf(buffer: Buffer): Promise<string> {
return new Promise((resolve) => { return new Promise((resolve) => {
const pdfParser = new PDFParser(null, true); const pdfParser = new PDFParser(null, true);
let completed = false; let completed = false;
const finish = (text: string) => { const finish = (text: string) => {
if (completed) return; if (completed) return;
completed = true; completed = true;
pdfParser.removeAllListeners(); pdfParser.removeAllListeners();
resolve(text); resolve(text);
}; };
pdfParser.on('pdfParser_dataError', () => finish('')); pdfParser.on('pdfParser_dataError', () => finish(''));
pdfParser.on('pdfParser_dataReady', () => pdfParser.on('pdfParser_dataReady', () => finish(pdfParser.getRawTextContent()));
finish(pdfParser.getRawTextContent())
);
try { try {
pdfParser.parseBuffer(buffer); pdfParser.parseBuffer(buffer);
} catch (err) { } catch (err) {
console.error('Error parsing PDF buffer', err); console.error('Error parsing PDF buffer', err);
finish(''); finish('');
} }
// Prevent hanging if the parser never emits events // Prevent hanging if the parser never emits events
setTimeout(() => finish(''), 10000); setTimeout(() => finish(''), 10000);
}); });
} }
export async function extractText( export async function extractText(buffer: Buffer, mimeType: string): Promise<string> {
buffer: Buffer, try {
mimeType: string if (mimeType === 'application/pdf') {
): Promise<string> { return await extractTextFromPdf(buffer);
try { }
if (mimeType === 'application/pdf') {
return await extractTextFromPdf(buffer);
}
if ( if (
mimeType === mimeType === 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'
'application/vnd.openxmlformats-officedocument.wordprocessingml.document' ) {
) { const { value } = await mammoth.extractRawText({ buffer });
const { value } = await mammoth.extractRawText({ buffer }); return value;
return value; }
}
if ( if (mimeType === 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet') {
mimeType === const workbook = xlsx.read(buffer, { type: 'buffer' });
'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' let fullText = '';
) { for (const sheetName of workbook.SheetNames) {
const workbook = xlsx.read(buffer, { type: 'buffer' }); const sheet = workbook.Sheets[sheetName];
let fullText = ''; const sheetText = xlsx.utils.sheet_to_txt(sheet);
for (const sheetName of workbook.SheetNames) { fullText += sheetText + '\n';
const sheet = workbook.Sheets[sheetName]; }
const sheetText = xlsx.utils.sheet_to_txt(sheet); return fullText;
fullText += sheetText + '\n'; }
}
return fullText;
}
if ( if (
mimeType.startsWith('text/') || mimeType.startsWith('text/') ||
mimeType === 'application/json' || mimeType === 'application/json' ||
mimeType === 'application/xml' mimeType === 'application/xml'
) { ) {
return buffer.toString('utf-8'); return buffer.toString('utf-8');
} }
} catch (error) { } catch (error) {
console.error( console.error(`Error extracting text from attachment with MIME type ${mimeType}:`, error);
`Error extracting text from attachment with MIME type ${mimeType}:`, return ''; // Return empty string on failure
error }
);
return ''; // Return empty string on failure
}
console.warn(`Unsupported MIME type for text extraction: ${mimeType}`); console.warn(`Unsupported MIME type for text extraction: ${mimeType}`);
return ''; // Return empty string for unsupported types return ''; // Return empty string for unsupported types
} }

72
packages/backend/src/iam-policy/iam-definitions.ts
View File

@@ -21,71 +21,67 @@
// =================================================================================== // ===================================================================================
const ARCHIVE_ACTIONS = { const ARCHIVE_ACTIONS = {
READ: 'archive:read', READ: 'archive:read',
SEARCH: 'archive:search', SEARCH: 'archive:search',
EXPORT: 'archive:export', EXPORT: 'archive:export',
} as const; } as const;
const ARCHIVE_RESOURCES = { const ARCHIVE_RESOURCES = {
ALL: 'archive/all', ALL: 'archive/all',
INGESTION_SOURCE: 'archive/ingestion-source/*', INGESTION_SOURCE: 'archive/ingestion-source/*',
MAILBOX: 'archive/mailbox/*', MAILBOX: 'archive/mailbox/*',
CUSTODIAN: 'archive/custodian/*', CUSTODIAN: 'archive/custodian/*',
} as const; } as const;
// =================================================================================== // ===================================================================================
// SERVICE: ingestion // SERVICE: ingestion
// =================================================================================== // ===================================================================================
const INGESTION_ACTIONS = { const INGESTION_ACTIONS = {
CREATE_SOURCE: 'ingestion:createSource', CREATE_SOURCE: 'ingestion:createSource',
READ_SOURCE: 'ingestion:readSource', READ_SOURCE: 'ingestion:readSource',
UPDATE_SOURCE: 'ingestion:updateSource', UPDATE_SOURCE: 'ingestion:updateSource',
DELETE_SOURCE: 'ingestion:deleteSource', DELETE_SOURCE: 'ingestion:deleteSource',
MANAGE_SYNC: 'ingestion:manageSync', // Covers triggering, pausing, and forcing syncs MANAGE_SYNC: 'ingestion:manageSync', // Covers triggering, pausing, and forcing syncs
} as const; } as const;
const INGESTION_RESOURCES = { const INGESTION_RESOURCES = {
ALL: 'ingestion-source/*', ALL: 'ingestion-source/*',
SOURCE: 'ingestion-source/{sourceId}', SOURCE: 'ingestion-source/{sourceId}',
} as const; } as const;
// =================================================================================== // ===================================================================================
// SERVICE: system // SERVICE: system
// =================================================================================== // ===================================================================================
const SYSTEM_ACTIONS = { const SYSTEM_ACTIONS = {
READ_SETTINGS: 'system:readSettings', READ_SETTINGS: 'system:readSettings',
UPDATE_SETTINGS: 'system:updateSettings', UPDATE_SETTINGS: 'system:updateSettings',
READ_USERS: 'system:readUsers', READ_USERS: 'system:readUsers',
CREATE_USER: 'system:createUser', CREATE_USER: 'system:createUser',
UPDATE_USER: 'system:updateUser', UPDATE_USER: 'system:updateUser',
DELETE_USER: 'system:deleteUser', DELETE_USER: 'system:deleteUser',
ASSIGN_ROLE: 'system:assignRole', ASSIGN_ROLE: 'system:assignRole',
} as const; } as const;
const SYSTEM_RESOURCES = { const SYSTEM_RESOURCES = {
SETTINGS: 'system/settings', SETTINGS: 'system/settings',
USERS: 'system/users', USERS: 'system/users',
USER: 'system/user/{userId}', USER: 'system/user/{userId}',
} as const; } as const;
// =================================================================================== // ===================================================================================
// SERVICE: dashboard // SERVICE: dashboard
// =================================================================================== // ===================================================================================
const DASHBOARD_ACTIONS = { const DASHBOARD_ACTIONS = {
READ: 'dashboard:read', READ: 'dashboard:read',
} as const; } as const;
const DASHBOARD_RESOURCES = { const DASHBOARD_RESOURCES = {
ALL: 'dashboard/*', ALL: 'dashboard/*',
} as const; } as const;
// =================================================================================== // ===================================================================================
// EXPORTED DEFINITIONS // EXPORTED DEFINITIONS
// =================================================================================== // ===================================================================================
@@ -95,10 +91,10 @@ const DASHBOARD_RESOURCES = {
* This is used by the policy validator to ensure that any action in a policy is recognized. * This is used by the policy validator to ensure that any action in a policy is recognized.
*/ */
export const ValidActions: Set<string> = new Set([ export const ValidActions: Set<string> = new Set([
...Object.values(ARCHIVE_ACTIONS), ...Object.values(ARCHIVE_ACTIONS),
...Object.values(INGESTION_ACTIONS), ...Object.values(INGESTION_ACTIONS),
...Object.values(SYSTEM_ACTIONS), ...Object.values(SYSTEM_ACTIONS),
...Object.values(DASHBOARD_ACTIONS), ...Object.values(DASHBOARD_ACTIONS),
]); ]);
/** /**
@@ -113,8 +109,8 @@ export const ValidActions: Set<string> = new Set([
* as is `archive/email/123-abc`. * as is `archive/email/123-abc`.
*/ */
export const ValidResourcePatterns = { export const ValidResourcePatterns = {
archive: /^archive\/(all|ingestion-source\/[^\/]+|mailbox\/[^\/]+|custodian\/[^\/]+)$/, archive: /^archive\/(all|ingestion-source\/[^\/]+|mailbox\/[^\/]+|custodian\/[^\/]+)$/,
ingestion: /^ingestion-source\/(\*|[^\/]+)$/, ingestion: /^ingestion-source\/(\*|[^\/]+)$/,
system: /^system\/(settings|users|user\/[^\/]+)$/, system: /^system\/(settings|users|user\/[^\/]+)$/,
dashboard: /^dashboard\/\*$/, dashboard: /^dashboard\/\*$/,
}; };

168
packages/backend/src/iam-policy/policy-validator.ts
View File

@@ -11,90 +11,96 @@ import { ValidActions, ValidResourcePatterns } from './iam-definitions';
* The verification logic is based on the centralized definitions in `iam-definitions.ts`. * The verification logic is based on the centralized definitions in `iam-definitions.ts`.
*/ */
export class PolicyValidator { export class PolicyValidator {
/** /**
* Validates a single policy statement to ensure its actions and resources are valid. * Validates a single policy statement to ensure its actions and resources are valid.
* *
* @param {PolicyStatement} statement - The policy statement to validate. * @param {PolicyStatement} statement - The policy statement to validate.
* @returns {{valid: boolean; reason?: string}} - An object containing a boolean `valid` property * @returns {{valid: boolean; reason?: string}} - An object containing a boolean `valid` property
* and an optional `reason` string if validation fails. * and an optional `reason` string if validation fails.
*/ */
public static isValid(statement: PolicyStatement): { valid: boolean; reason: string; } { public static isValid(statement: PolicyStatement): { valid: boolean; reason: string } {
if (!statement || !statement.Action || !statement.Resource || !statement.Effect) { if (!statement || !statement.Action || !statement.Resource || !statement.Effect) {
return { valid: false, reason: 'Policy statement is missing required fields.' }; return { valid: false, reason: 'Policy statement is missing required fields.' };
} }
// 1. Validate Actions // 1. Validate Actions
for (const action of statement.Action) { for (const action of statement.Action) {
const { valid, reason } = this.isActionValid(action); const { valid, reason } = this.isActionValid(action);
if (!valid) { if (!valid) {
return { valid: false, reason }; return { valid: false, reason };
} }
} }
// 2. Validate Resources // 2. Validate Resources
for (const resource of statement.Resource) { for (const resource of statement.Resource) {
const { valid, reason } = this.isResourceValid(resource); const { valid, reason } = this.isResourceValid(resource);
if (!valid) { if (!valid) {
return { valid: false, reason }; return { valid: false, reason };
} }
} }
return { valid: true, reason: 'valid' }; return { valid: true, reason: 'valid' };
} }
/** /**
* Checks if a single action string is valid. * Checks if a single action string is valid.
* *
* Logic: * Logic:
* - If the action contains a wildcard (e.g., 'archive:*'), it checks if the service part * - If the action contains a wildcard (e.g., 'archive:*'), it checks if the service part
* (e.g., 'archive') is a recognized service. * (e.g., 'archive') is a recognized service.
* - If there is no wildcard, it checks if the full action string (e.g., 'archive:read') * - If there is no wildcard, it checks if the full action string (e.g., 'archive:read')
* exists in the `ValidActions` set. * exists in the `ValidActions` set.
* *
* @param {string} action - The action string to validate. * @param {string} action - The action string to validate.
* @returns {{valid: boolean; reason?: string}} - An object indicating validity and a reason for failure. * @returns {{valid: boolean; reason?: string}} - An object indicating validity and a reason for failure.
*/ */
private static isActionValid(action: string): { valid: boolean; reason: string; } { private static isActionValid(action: string): { valid: boolean; reason: string } {
if (action === '*') { if (action === '*') {
return { valid: true, reason: 'valid' }; return { valid: true, reason: 'valid' };
} }
if (action.endsWith(':*')) { if (action.endsWith(':*')) {
const service = action.split(':')[0]; const service = action.split(':')[0];
if (service in ValidResourcePatterns) { if (service in ValidResourcePatterns) {
return { valid: true, reason: 'valid' }; return { valid: true, reason: 'valid' };
} }
return { valid: false, reason: `Invalid service '${service}' in action wildcard '${action}'.` }; return {
} valid: false,
if (ValidActions.has(action)) { reason: `Invalid service '${service}' in action wildcard '${action}'.`,
return { valid: true, reason: 'valid' }; };
} }
return { valid: false, reason: `Action '${action}' is not a valid action.` }; if (ValidActions.has(action)) {
} return { valid: true, reason: 'valid' };
}
return { valid: false, reason: `Action '${action}' is not a valid action.` };
}
/** /**
* Checks if a single resource string has a valid format. * Checks if a single resource string has a valid format.
* *
* Logic: * Logic:
* - It extracts the service name from the resource string (e.g., 'archive' from 'archive/all'). * - It extracts the service name from the resource string (e.g., 'archive' from 'archive/all').
* - It looks up the corresponding regular expression for that service in `ValidResourcePatterns`. * - It looks up the corresponding regular expression for that service in `ValidResourcePatterns`.
* - It tests the resource string against the pattern. If the service does not exist or the * - It tests the resource string against the pattern. If the service does not exist or the
* pattern does not match, the resource is considered invalid. * pattern does not match, the resource is considered invalid.
* *
* @param {string} resource - The resource string to validate. * @param {string} resource - The resource string to validate.
* @returns {{valid: boolean; reason?: string}} - An object indicating validity and a reason for failure. * @returns {{valid: boolean; reason?: string}} - An object indicating validity and a reason for failure.
*/ */
private static isResourceValid(resource: string): { valid: boolean; reason: string; } { private static isResourceValid(resource: string): { valid: boolean; reason: string } {
const service = resource.split('/')[0]; const service = resource.split('/')[0];
if (service === '*') { if (service === '*') {
return { valid: true, reason: 'valid' }; return { valid: true, reason: 'valid' };
} }
if (service in ValidResourcePatterns) { if (service in ValidResourcePatterns) {
const pattern = ValidResourcePatterns[service as keyof typeof ValidResourcePatterns]; const pattern = ValidResourcePatterns[service as keyof typeof ValidResourcePatterns];
if (pattern.test(resource)) { if (pattern.test(resource)) {
return { valid: true, reason: 'valid' }; return { valid: true, reason: 'valid' };
} }
return { valid: false, reason: `Resource '${resource}' does not match the expected format for the '${service}' service.` }; return {
} valid: false,
return { valid: false, reason: `Invalid service '${service}' in resource '${resource}'.` }; reason: `Resource '${resource}' does not match the expected format for the '${service}' service.`,
} };
}
return { valid: false, reason: `Invalid service '${service}' in resource '${resource}'.` };
}
} }

47
packages/backend/src/index.ts
View File

@@ -22,21 +22,16 @@ import { IamService } from './services/IamService';
import { StorageService } from './services/StorageService'; import { StorageService } from './services/StorageService';
import { SearchService } from './services/SearchService'; import { SearchService } from './services/SearchService';
// Load environment variables // Load environment variables
dotenv.config(); dotenv.config();
// --- Environment Variable Validation --- // --- Environment Variable Validation ---
const { const { PORT_BACKEND, JWT_SECRET, JWT_EXPIRES_IN } = process.env;
PORT_BACKEND,
JWT_SECRET,
JWT_EXPIRES_IN
} = process.env;
if (!PORT_BACKEND || !JWT_SECRET || !JWT_EXPIRES_IN) { if (!PORT_BACKEND || !JWT_SECRET || !JWT_EXPIRES_IN) {
throw new Error('Missing required environment variables for the backend: PORT_BACKEND, JWT_SECRET, JWT_EXPIRES_IN.'); throw new Error(
'Missing required environment variables for the backend: PORT_BACKEND, JWT_SECRET, JWT_EXPIRES_IN.'
);
} }
// --- Dependency Injection Setup --- // --- Dependency Injection Setup ---
@@ -83,30 +78,30 @@ app.use('/v1/test', testRouter);
// Example of a protected route // Example of a protected route
app.get('/v1/protected', requireAuth(authService), (req, res) => { app.get('/v1/protected', requireAuth(authService), (req, res) => {
res.json({ res.json({
message: 'You have accessed a protected route!', message: 'You have accessed a protected route!',
user: req.user // The user payload is attached by the requireAuth middleware user: req.user, // The user payload is attached by the requireAuth middleware
}); });
}); });
app.get("/", (req, res) => { app.get('/', (req, res) => {
res.send('Backend is running!'); res.send('Backend is running!');
}); });
// --- Server Start --- // --- Server Start ---
const startServer = async () => { const startServer = async () => {
try { try {
// Configure the Meilisearch index on startup // Configure the Meilisearch index on startup
console.log('Configuring email index...'); console.log('Configuring email index...');
await searchService.configureEmailIndex(); await searchService.configureEmailIndex();
app.listen(PORT_BACKEND, () => { app.listen(PORT_BACKEND, () => {
console.log(`Backend listening at http://localhost:${PORT_BACKEND}`); console.log(`Backend listening at http://localhost:${PORT_BACKEND}`);
}); });
} catch (error) { } catch (error) {
console.error('Failed to start the server:', error); console.error('Failed to start the server:', error);
process.exit(1); process.exit(1);
} }
}; };
startServer(); startServer();

134
packages/backend/src/jobs/processors/continuous-sync.processor.ts
View File

@@ -6,74 +6,80 @@ import { flowProducer } from '../queues';
import { logger } from '../../config/logger'; import { logger } from '../../config/logger';
export default async (job: Job<IContinuousSyncJob>) => { export default async (job: Job<IContinuousSyncJob>) => {
const { ingestionSourceId } = job.data; const { ingestionSourceId } = job.data;
logger.info({ ingestionSourceId }, 'Starting continuous sync job.'); logger.info({ ingestionSourceId }, 'Starting continuous sync job.');
const source = await IngestionService.findById(ingestionSourceId); const source = await IngestionService.findById(ingestionSourceId);
if (!source || !['error', 'active'].includes(source.status)) { if (!source || !['error', 'active'].includes(source.status)) {
logger.warn({ ingestionSourceId, status: source?.status }, 'Skipping continuous sync for non-active or non-error source.'); logger.warn(
return; { ingestionSourceId, status: source?.status },
} 'Skipping continuous sync for non-active or non-error source.'
);
return;
}
await IngestionService.update(ingestionSourceId, { await IngestionService.update(ingestionSourceId, {
status: 'syncing', status: 'syncing',
lastSyncStartedAt: new Date(), lastSyncStartedAt: new Date(),
}); });
const connector = EmailProviderFactory.createConnector(source); const connector = EmailProviderFactory.createConnector(source);
try { try {
const jobs = []; const jobs = [];
for await (const user of connector.listAllUsers()) { for await (const user of connector.listAllUsers()) {
if (user.primaryEmail) { if (user.primaryEmail) {
jobs.push({ jobs.push({
name: 'process-mailbox', name: 'process-mailbox',
queueName: 'ingestion', queueName: 'ingestion',
data: { data: {
ingestionSourceId: source.id, ingestionSourceId: source.id,
userEmail: user.primaryEmail userEmail: user.primaryEmail,
}, },
opts: { opts: {
removeOnComplete: { removeOnComplete: {
age: 60 * 10 // 10 minutes age: 60 * 10, // 10 minutes
}, },
removeOnFail: { removeOnFail: {
age: 60 * 30 // 30 minutes age: 60 * 30, // 30 minutes
}, },
timeout: 1000 * 60 * 30 // 30 minutes timeout: 1000 * 60 * 30, // 30 minutes
} },
}); });
} }
} }
// } // }
if (jobs.length > 0) { if (jobs.length > 0) {
await flowProducer.add({ await flowProducer.add({
name: 'sync-cycle-finished', name: 'sync-cycle-finished',
queueName: 'ingestion', queueName: 'ingestion',
data: { data: {
ingestionSourceId, ingestionSourceId,
isInitialImport: false isInitialImport: false,
}, },
children: jobs, children: jobs,
opts: { opts: {
removeOnComplete: true, removeOnComplete: true,
removeOnFail: true removeOnFail: true,
} },
}); });
} }
// The status will be set back to 'active' by the 'sync-cycle-finished' job // The status will be set back to 'active' by the 'sync-cycle-finished' job
// once all the mailboxes have been processed. // once all the mailboxes have been processed.
logger.info({ ingestionSourceId }, 'Continuous sync job finished dispatching mailbox jobs.'); logger.info(
{ ingestionSourceId },
} catch (error) { 'Continuous sync job finished dispatching mailbox jobs.'
logger.error({ err: error, ingestionSourceId }, 'Continuous sync job failed.'); );
await IngestionService.update(ingestionSourceId, { } catch (error) {
status: 'error', logger.error({ err: error, ingestionSourceId }, 'Continuous sync job failed.');
lastSyncFinishedAt: new Date(), await IngestionService.update(ingestionSourceId, {
lastSyncStatusMessage: error instanceof Error ? error.message : 'An unknown error occurred during sync.', status: 'error',
}); lastSyncFinishedAt: new Date(),
throw error; lastSyncStatusMessage:
} error instanceof Error ? error.message : 'An unknown error occurred during sync.',
});
throw error;
}
}; };

8
packages/backend/src/jobs/processors/index-email.processor.ts
View File

@@ -9,8 +9,8 @@ const storageService = new StorageService();
const databaseService = new DatabaseService(); const databaseService = new DatabaseService();
const indexingService = new IndexingService(databaseService, searchService, storageService); const indexingService = new IndexingService(databaseService, searchService, storageService);
export default async function (job: Job<{ emailId: string; }>) { export default async function (job: Job<{ emailId: string }>) {
const { emailId } = job.data; const { emailId } = job.data;
console.log(`Indexing email with ID: ${emailId}`); console.log(`Indexing email with ID: ${emailId}`);
await indexingService.indexEmailById(emailId); await indexingService.indexEmailById(emailId);
} }

152
packages/backend/src/jobs/processors/initial-import.processor.ts
View File

@@ -5,85 +5,89 @@ import { EmailProviderFactory } from '../../services/EmailProviderFactory';
import { flowProducer } from '../queues'; import { flowProducer } from '../queues';
import { logger } from '../../config/logger'; import { logger } from '../../config/logger';
export default async (job: Job<IInitialImportJob>) => { export default async (job: Job<IInitialImportJob>) => {
const { ingestionSourceId } = job.data; const { ingestionSourceId } = job.data;
logger.info({ ingestionSourceId }, 'Starting initial import master job'); logger.info({ ingestionSourceId }, 'Starting initial import master job');
try { try {
const source = await IngestionService.findById(ingestionSourceId); const source = await IngestionService.findById(ingestionSourceId);
if (!source) { if (!source) {
throw new Error(`Ingestion source with ID ${ingestionSourceId} not found`); throw new Error(`Ingestion source with ID ${ingestionSourceId} not found`);
} }
await IngestionService.update(ingestionSourceId, { await IngestionService.update(ingestionSourceId, {
status: 'importing', status: 'importing',
lastSyncStatusMessage: 'Starting initial import...' lastSyncStatusMessage: 'Starting initial import...',
}); });
const connector = EmailProviderFactory.createConnector(source); const connector = EmailProviderFactory.createConnector(source);
// if (connector instanceof GoogleWorkspaceConnector || connector instanceof MicrosoftConnector) { // if (connector instanceof GoogleWorkspaceConnector || connector instanceof MicrosoftConnector) {
const jobs: FlowChildJob[] = []; const jobs: FlowChildJob[] = [];
let userCount = 0; let userCount = 0;
for await (const user of connector.listAllUsers()) { for await (const user of connector.listAllUsers()) {
if (user.primaryEmail) { if (user.primaryEmail) {
jobs.push({ jobs.push({
name: 'process-mailbox', name: 'process-mailbox',
queueName: 'ingestion', queueName: 'ingestion',
data: { data: {
ingestionSourceId, ingestionSourceId,
userEmail: user.primaryEmail, userEmail: user.primaryEmail,
}, },
opts: { opts: {
removeOnComplete: { removeOnComplete: {
age: 60 * 10 // 10 minutes age: 60 * 10, // 10 minutes
}, },
removeOnFail: { removeOnFail: {
age: 60 * 30 // 30 minutes age: 60 * 30, // 30 minutes
}, },
attempts: 1, attempts: 1,
// failParentOnFailure: true // failParentOnFailure: true
} },
}); });
userCount++; userCount++;
} }
} }
if (jobs.length > 0) { if (jobs.length > 0) {
logger.info({ ingestionSourceId, userCount }, 'Adding sync-cycle-finished job to the queue'); logger.info(
await flowProducer.add({ { ingestionSourceId, userCount },
name: 'sync-cycle-finished', 'Adding sync-cycle-finished job to the queue'
queueName: 'ingestion', );
data: { await flowProducer.add({
ingestionSourceId, name: 'sync-cycle-finished',
userCount, queueName: 'ingestion',
isInitialImport: true data: {
}, ingestionSourceId,
children: jobs, userCount,
opts: { isInitialImport: true,
removeOnComplete: true, },
removeOnFail: true children: jobs,
} opts: {
}); removeOnComplete: true,
} else { removeOnFail: true,
const fileBasedIngestions = IngestionService.returnFileBasedIngestions(); },
const finalStatus = fileBasedIngestions.includes(source.provider) ? 'imported' : 'active'; });
// If there are no users, we can consider the import finished and set to active } else {
await IngestionService.update(ingestionSourceId, { const fileBasedIngestions = IngestionService.returnFileBasedIngestions();
status: finalStatus, const finalStatus = fileBasedIngestions.includes(source.provider)
lastSyncFinishedAt: new Date(), ? 'imported'
lastSyncStatusMessage: 'Initial import complete. No users found.' : 'active';
}); // If there are no users, we can consider the import finished and set to active
} await IngestionService.update(ingestionSourceId, {
status: finalStatus,
lastSyncFinishedAt: new Date(),
lastSyncStatusMessage: 'Initial import complete. No users found.',
});
}
logger.info({ ingestionSourceId }, 'Finished initial import master job'); logger.info({ ingestionSourceId }, 'Finished initial import master job');
} catch (error) { } catch (error) {
logger.error({ err: error, ingestionSourceId }, 'Error in initial import master job'); logger.error({ err: error, ingestionSourceId }, 'Error in initial import master job');
await IngestionService.update(ingestionSourceId, { await IngestionService.update(ingestionSourceId, {
status: 'error', status: 'error',
lastSyncStatusMessage: `Initial import failed: ${error instanceof Error ? error.message : 'Unknown error'}` lastSyncStatusMessage: `Initial import failed: ${error instanceof Error ? error.message : 'Unknown error'}`,
}); });
throw error; throw error;
} }
}; };

58
packages/backend/src/jobs/processors/process-mailbox.processor.ts
View File

@@ -14,40 +14,40 @@ import { StorageService } from '../../services/StorageService';
* 'process-mailbox' jobs, aggregating successes, and reporting detailed failures. * 'process-mailbox' jobs, aggregating successes, and reporting detailed failures.
*/ */
export const processMailboxProcessor = async (job: Job<IProcessMailboxJob, SyncState, string>) => { export const processMailboxProcessor = async (job: Job<IProcessMailboxJob, SyncState, string>) => {
const { ingestionSourceId, userEmail } = job.data; const { ingestionSourceId, userEmail } = job.data;
logger.info({ ingestionSourceId, userEmail }, `Processing mailbox for user`); logger.info({ ingestionSourceId, userEmail }, `Processing mailbox for user`);
try { try {
const source = await IngestionService.findById(ingestionSourceId); const source = await IngestionService.findById(ingestionSourceId);
if (!source) { if (!source) {
throw new Error(`Ingestion source with ID ${ingestionSourceId} not found`); throw new Error(`Ingestion source with ID ${ingestionSourceId} not found`);
} }
const connector = EmailProviderFactory.createConnector(source); const connector = EmailProviderFactory.createConnector(source);
const ingestionService = new IngestionService(); const ingestionService = new IngestionService();
const storageService = new StorageService(); const storageService = new StorageService();
// Pass the sync state for the entire source, the connector will handle per-user logic if necessary // Pass the sync state for the entire source, the connector will handle per-user logic if necessary
for await (const email of connector.fetchEmails(userEmail, source.syncState)) { for await (const email of connector.fetchEmails(userEmail, source.syncState)) {
if (email) { if (email) {
await ingestionService.processEmail(email, source, storageService, userEmail); await ingestionService.processEmail(email, source, storageService, userEmail);
} }
} }
const newSyncState = connector.getUpdatedSyncState(userEmail); const newSyncState = connector.getUpdatedSyncState(userEmail);
logger.info({ ingestionSourceId, userEmail }, `Finished processing mailbox for user`); logger.info({ ingestionSourceId, userEmail }, `Finished processing mailbox for user`);
// Return the new sync state to be aggregated by the parent flow // Return the new sync state to be aggregated by the parent flow
return newSyncState; return newSyncState;
} catch (error) { } catch (error) {
logger.error({ err: error, ingestionSourceId, userEmail }, 'Error processing mailbox'); logger.error({ err: error, ingestionSourceId, userEmail }, 'Error processing mailbox');
const errorMessage = error instanceof Error ? error.message : 'An unknown error occurred'; const errorMessage = error instanceof Error ? error.message : 'An unknown error occurred';
const processMailboxError: ProcessMailboxError = { const processMailboxError: ProcessMailboxError = {
error: true, error: true,
message: `Failed to process mailbox for ${userEmail}: ${errorMessage}` message: `Failed to process mailbox for ${userEmail}: ${errorMessage}`,
}; };
return processMailboxError; return processMailboxError;
} }
}; };

27
packages/backend/src/jobs/processors/schedule-continuous-sync.processor.ts
View File

@@ -5,22 +5,15 @@ import { or, eq } from 'drizzle-orm';
import { ingestionQueue } from '../queues'; import { ingestionQueue } from '../queues';
export default async (job: Job) => { export default async (job: Job) => {
console.log( console.log('Scheduler running: Looking for active or error ingestion sources to sync.');
'Scheduler running: Looking for active or error ingestion sources to sync.' // find all sources that have the status of active or error for continuous syncing.
); const sourcesToSync = await db
// find all sources that have the status of active or error for continuous syncing. .select({ id: ingestionSources.id })
const sourcesToSync = await db .from(ingestionSources)
.select({ id: ingestionSources.id }) .where(or(eq(ingestionSources.status, 'active'), eq(ingestionSources.status, 'error')));
.from(ingestionSources)
.where(
or(
eq(ingestionSources.status, 'active'),
eq(ingestionSources.status, 'error')
)
);
for (const source of sourcesToSync) { for (const source of sourcesToSync) {
// The status field on the ingestion source is used to prevent duplicate syncs. // The status field on the ingestion source is used to prevent duplicate syncs.
await ingestionQueue.add('continuous-sync', { ingestionSourceId: source.id }); await ingestionQueue.add('continuous-sync', { ingestionSourceId: source.id });
} }
}; };

129
packages/backend/src/jobs/processors/sync-cycle-finished.processor.ts
View File

@@ -1,16 +1,21 @@
import { Job } from 'bullmq'; import { Job } from 'bullmq';
import { IngestionService } from '../../services/IngestionService'; import { IngestionService } from '../../services/IngestionService';
import { logger } from '../../config/logger'; import { logger } from '../../config/logger';
import { SyncState, ProcessMailboxError, IngestionStatus, IngestionProvider } from '@open-archiver/types'; import {
SyncState,
ProcessMailboxError,
IngestionStatus,
IngestionProvider,
} from '@open-archiver/types';
import { db } from '../../database'; import { db } from '../../database';
import { ingestionSources } from '../../database/schema'; import { ingestionSources } from '../../database/schema';
import { eq } from 'drizzle-orm'; import { eq } from 'drizzle-orm';
import { deepmerge } from 'deepmerge-ts'; import { deepmerge } from 'deepmerge-ts';
interface ISyncCycleFinishedJob { interface ISyncCycleFinishedJob {
ingestionSourceId: string; ingestionSourceId: string;
userCount?: number; // Optional, as it's only relevant for the initial import userCount?: number; // Optional, as it's only relevant for the initial import
isInitialImport: boolean; isInitialImport: boolean;
} }
/** /**
@@ -28,63 +33,75 @@ interface ISyncCycleFinishedJob {
* *
*/ */
export default async (job: Job<ISyncCycleFinishedJob, any, string>) => { export default async (job: Job<ISyncCycleFinishedJob, any, string>) => {
const { ingestionSourceId, userCount, isInitialImport } = job.data; const { ingestionSourceId, userCount, isInitialImport } = job.data;
logger.info({ ingestionSourceId, userCount, isInitialImport }, 'Sync cycle finished job started'); logger.info(
{ ingestionSourceId, userCount, isInitialImport },
'Sync cycle finished job started'
);
try { try {
const childrenValues = await job.getChildrenValues<SyncState | ProcessMailboxError>(); const childrenValues = await job.getChildrenValues<SyncState | ProcessMailboxError>();
const allChildJobs = Object.values(childrenValues); const allChildJobs = Object.values(childrenValues);
// if data has error property, it is a failed job // if data has error property, it is a failed job
const failedJobs = allChildJobs.filter(v => v && (v as any).error) as ProcessMailboxError[]; const failedJobs = allChildJobs.filter(
// if data doesn't have error property, it is a successful job with SyncState (v) => v && (v as any).error
const successfulJobs = allChildJobs.filter(v => !v || !(v as any).error) as SyncState[]; ) as ProcessMailboxError[];
// if data doesn't have error property, it is a successful job with SyncState
const successfulJobs = allChildJobs.filter((v) => !v || !(v as any).error) as SyncState[];
const finalSyncState = deepmerge(...successfulJobs.filter(s => s && Object.keys(s).length > 0)); const finalSyncState = deepmerge(
...successfulJobs.filter((s) => s && Object.keys(s).length > 0)
);
const source = await IngestionService.findById(ingestionSourceId); const source = await IngestionService.findById(ingestionSourceId);
let status: IngestionStatus = 'active'; let status: IngestionStatus = 'active';
const fileBasedIngestions = IngestionService.returnFileBasedIngestions(); const fileBasedIngestions = IngestionService.returnFileBasedIngestions();
if (fileBasedIngestions.includes(source.provider)) { if (fileBasedIngestions.includes(source.provider)) {
status = 'imported'; status = 'imported';
} }
let message: string; let message: string;
// Check for a specific rate-limit message from the successful jobs // Check for a specific rate-limit message from the successful jobs
const rateLimitMessage = successfulJobs.find(j => j.statusMessage)?.statusMessage; const rateLimitMessage = successfulJobs.find((j) => j.statusMessage)?.statusMessage;
if (failedJobs.length > 0) { if (failedJobs.length > 0) {
status = 'error'; status = 'error';
const errorMessages = failedJobs.map(j => j.message).join('\n'); const errorMessages = failedJobs.map((j) => j.message).join('\n');
message = `Sync cycle completed with ${failedJobs.length} error(s):\n${errorMessages}`; message = `Sync cycle completed with ${failedJobs.length} error(s):\n${errorMessages}`;
logger.error({ ingestionSourceId, errors: errorMessages }, 'Sync cycle finished with errors.'); logger.error(
} else if (rateLimitMessage) { { ingestionSourceId, errors: errorMessages },
message = rateLimitMessage; 'Sync cycle finished with errors.'
logger.warn({ ingestionSourceId, message }, 'Sync cycle paused due to rate limiting.'); );
} } else if (rateLimitMessage) {
else { message = rateLimitMessage;
message = 'Continuous sync cycle finished successfully.'; logger.warn({ ingestionSourceId, message }, 'Sync cycle paused due to rate limiting.');
if (isInitialImport) { } else {
message = `Initial import finished for ${userCount} mailboxes.`; message = 'Continuous sync cycle finished successfully.';
} if (isInitialImport) {
logger.info({ ingestionSourceId }, 'Successfully updated status and final sync state.'); message = `Initial import finished for ${userCount} mailboxes.`;
} }
logger.info({ ingestionSourceId }, 'Successfully updated status and final sync state.');
}
await db await db
.update(ingestionSources) .update(ingestionSources)
.set({ .set({
status, status,
lastSyncFinishedAt: new Date(), lastSyncFinishedAt: new Date(),
lastSyncStatusMessage: message, lastSyncStatusMessage: message,
syncState: finalSyncState syncState: finalSyncState,
}) })
.where(eq(ingestionSources.id, ingestionSourceId)); .where(eq(ingestionSources.id, ingestionSourceId));
} catch (error) { } catch (error) {
logger.error({ err: error, ingestionSourceId }, 'An unexpected error occurred while finalizing the sync cycle.'); logger.error(
await IngestionService.update(ingestionSourceId, { { err: error, ingestionSourceId },
status: 'error', 'An unexpected error occurred while finalizing the sync cycle.'
lastSyncFinishedAt: new Date(), );
lastSyncStatusMessage: 'An unexpected error occurred while finalizing the sync cycle.' await IngestionService.update(ingestionSourceId, {
}); status: 'error',
} lastSyncFinishedAt: new Date(),
lastSyncStatusMessage: 'An unexpected error occurred while finalizing the sync cycle.',
});
}
}; };

30
packages/backend/src/jobs/queues.ts
View File

@@ -5,25 +5,25 @@ export const flowProducer = new FlowProducer({ connection });
// Default job options // Default job options
const defaultJobOptions = { const defaultJobOptions = {
attempts: 5, attempts: 5,
backoff: { backoff: {
type: 'exponential', type: 'exponential',
delay: 1000 delay: 1000,
}, },
removeOnComplete: { removeOnComplete: {
count: 1000 count: 1000,
}, },
removeOnFail: { removeOnFail: {
count: 5000 count: 5000,
} },
}; };
export const ingestionQueue = new Queue('ingestion', { export const ingestionQueue = new Queue('ingestion', {
connection, connection,
defaultJobOptions defaultJobOptions,
}); });
export const indexingQueue = new Queue('indexing', { export const indexingQueue = new Queue('indexing', {
connection, connection,
defaultJobOptions defaultJobOptions,
}); });

22
packages/backend/src/jobs/schedulers/sync-scheduler.ts
View File

@@ -3,18 +3,18 @@ import { ingestionQueue } from '../queues';
import { config } from '../../config'; import { config } from '../../config';
const scheduleContinuousSync = async () => { const scheduleContinuousSync = async () => {
// This job will run every 15 minutes // This job will run every 15 minutes
await ingestionQueue.add( await ingestionQueue.add(
'schedule-continuous-sync', 'schedule-continuous-sync',
{}, {},
{ {
repeat: { repeat: {
pattern: config.app.syncFrequency pattern: config.app.syncFrequency,
}, },
} }
); );
}; };
scheduleContinuousSync().then(() => { scheduleContinuousSync().then(() => {
console.log('Continuous sync scheduler started.'); console.log('Continuous sync scheduler started.');
}); });

346
packages/backend/src/services/ArchivedEmailService.ts
View File

@@ -2,215 +2,213 @@ import { count, desc, eq, asc, and } from 'drizzle-orm';
import { db } from '../database'; import { db } from '../database';
import { archivedEmails, attachments, emailAttachments } from '../database/schema'; import { archivedEmails, attachments, emailAttachments } from '../database/schema';
import type { import type {
PaginatedArchivedEmails, PaginatedArchivedEmails,
ArchivedEmail, ArchivedEmail,
Recipient, Recipient,
ThreadEmail, ThreadEmail,
} from '@open-archiver/types'; } from '@open-archiver/types';
import { StorageService } from './StorageService'; import { StorageService } from './StorageService';
import { SearchService } from './SearchService'; import { SearchService } from './SearchService';
import type { Readable } from 'stream'; import type { Readable } from 'stream';
interface DbRecipients { interface DbRecipients {
to: { name: string; address: string; }[]; to: { name: string; address: string }[];
cc: { name: string; address: string; }[]; cc: { name: string; address: string }[];
bcc: { name: string; address: string; }[]; bcc: { name: string; address: string }[];
} }
async function streamToBuffer(stream: Readable): Promise<Buffer> { async function streamToBuffer(stream: Readable): Promise<Buffer> {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
const chunks: Buffer[] = []; const chunks: Buffer[] = [];
stream.on('data', (chunk) => chunks.push(chunk)); stream.on('data', (chunk) => chunks.push(chunk));
stream.on('error', reject); stream.on('error', reject);
stream.on('end', () => resolve(Buffer.concat(chunks))); stream.on('end', () => resolve(Buffer.concat(chunks)));
}); });
} }
export class ArchivedEmailService { export class ArchivedEmailService {
private static mapRecipients(dbRecipients: unknown): Recipient[] { private static mapRecipients(dbRecipients: unknown): Recipient[] {
const { to = [], cc = [], bcc = [] } = dbRecipients as DbRecipients; const { to = [], cc = [], bcc = [] } = dbRecipients as DbRecipients;
const allRecipients = [...to, ...cc, ...bcc]; const allRecipients = [...to, ...cc, ...bcc];
return allRecipients.map((r) => ({ return allRecipients.map((r) => ({
name: r.name, name: r.name,
email: r.address email: r.address,
})); }));
} }
public static async getArchivedEmails( public static async getArchivedEmails(
ingestionSourceId: string, ingestionSourceId: string,
page: number, page: number,
limit: number limit: number
): Promise<PaginatedArchivedEmails> { ): Promise<PaginatedArchivedEmails> {
const offset = (page - 1) * limit; const offset = (page - 1) * limit;
const [total] = await db const [total] = await db
.select({ .select({
count: count(archivedEmails.id) count: count(archivedEmails.id),
}) })
.from(archivedEmails) .from(archivedEmails)
.where(eq(archivedEmails.ingestionSourceId, ingestionSourceId)); .where(eq(archivedEmails.ingestionSourceId, ingestionSourceId));
const items = await db const items = await db
.select() .select()
.from(archivedEmails) .from(archivedEmails)
.where(eq(archivedEmails.ingestionSourceId, ingestionSourceId)) .where(eq(archivedEmails.ingestionSourceId, ingestionSourceId))
.orderBy(desc(archivedEmails.sentAt)) .orderBy(desc(archivedEmails.sentAt))
.limit(limit) .limit(limit)
.offset(offset); .offset(offset);
return { return {
items: items.map((item) => ({ items: items.map((item) => ({
...item, ...item,
recipients: this.mapRecipients(item.recipients), recipients: this.mapRecipients(item.recipients),
tags: (item.tags as string[] | null) || null, tags: (item.tags as string[] | null) || null,
path: item.path || null path: item.path || null,
})), })),
total: total.count, total: total.count,
page, page,
limit limit,
}; };
} }
public static async getArchivedEmailById(emailId: string): Promise<ArchivedEmail | null> { public static async getArchivedEmailById(emailId: string): Promise<ArchivedEmail | null> {
const [email] = await db const [email] = await db
.select() .select()
.from(archivedEmails) .from(archivedEmails)
.where(eq(archivedEmails.id, emailId)); .where(eq(archivedEmails.id, emailId));
if (!email) { if (!email) {
return null; return null;
} }
let threadEmails: ThreadEmail[] = []; let threadEmails: ThreadEmail[] = [];
if (email.threadId) { if (email.threadId) {
threadEmails = await db.query.archivedEmails.findMany({ threadEmails = await db.query.archivedEmails.findMany({
where: and( where: and(
eq(archivedEmails.threadId, email.threadId), eq(archivedEmails.threadId, email.threadId),
eq(archivedEmails.ingestionSourceId, email.ingestionSourceId) eq(archivedEmails.ingestionSourceId, email.ingestionSourceId)
), ),
orderBy: [asc(archivedEmails.sentAt)], orderBy: [asc(archivedEmails.sentAt)],
columns: { columns: {
id: true, id: true,
subject: true, subject: true,
sentAt: true, sentAt: true,
senderEmail: true, senderEmail: true,
}, },
}); });
} }
const storage = new StorageService(); const storage = new StorageService();
const rawStream = await storage.get(email.storagePath); const rawStream = await storage.get(email.storagePath);
const raw = await streamToBuffer(rawStream as Readable); const raw = await streamToBuffer(rawStream as Readable);
const mappedEmail = { const mappedEmail = {
...email, ...email,
recipients: this.mapRecipients(email.recipients), recipients: this.mapRecipients(email.recipients),
raw, raw,
thread: threadEmails, thread: threadEmails,
tags: (email.tags as string[] | null) || null, tags: (email.tags as string[] | null) || null,
path: email.path || null path: email.path || null,
}; };
if (email.hasAttachments) { if (email.hasAttachments) {
const emailAttachmentsResult = await db const emailAttachmentsResult = await db
.select({ .select({
id: attachments.id, id: attachments.id,
filename: attachments.filename, filename: attachments.filename,
mimeType: attachments.mimeType, mimeType: attachments.mimeType,
sizeBytes: attachments.sizeBytes, sizeBytes: attachments.sizeBytes,
storagePath: attachments.storagePath storagePath: attachments.storagePath,
}) })
.from(emailAttachments) .from(emailAttachments)
.innerJoin(attachments, eq(emailAttachments.attachmentId, attachments.id)) .innerJoin(attachments, eq(emailAttachments.attachmentId, attachments.id))
.where(eq(emailAttachments.emailId, emailId)); .where(eq(emailAttachments.emailId, emailId));
// const attachmentsWithRaw = await Promise.all( // const attachmentsWithRaw = await Promise.all(
// emailAttachmentsResult.map(async (attachment) => { // emailAttachmentsResult.map(async (attachment) => {
// const rawStream = await storage.get(attachment.storagePath); // const rawStream = await storage.get(attachment.storagePath);
// const raw = await streamToBuffer(rawStream as Readable); // const raw = await streamToBuffer(rawStream as Readable);
// return { ...attachment, raw }; // return { ...attachment, raw };
// }) // })
// ); // );
return { return {
...mappedEmail, ...mappedEmail,
attachments: emailAttachmentsResult attachments: emailAttachmentsResult,
}; };
} }
return mappedEmail; return mappedEmail;
} }
public static async deleteArchivedEmail(emailId: string): Promise<void> { public static async deleteArchivedEmail(emailId: string): Promise<void> {
const [email] = await db const [email] = await db
.select() .select()
.from(archivedEmails) .from(archivedEmails)
.where(eq(archivedEmails.id, emailId)); .where(eq(archivedEmails.id, emailId));
if (!email) { if (!email) {
throw new Error('Archived email not found'); throw new Error('Archived email not found');
} }
const storage = new StorageService(); const storage = new StorageService();
// Load and handle attachments before deleting the email itself // Load and handle attachments before deleting the email itself
if (email.hasAttachments) { if (email.hasAttachments) {
const emailAttachmentsResult = await db const emailAttachmentsResult = await db
.select({ .select({
attachmentId: attachments.id, attachmentId: attachments.id,
storagePath: attachments.storagePath, storagePath: attachments.storagePath,
}) })
.from(emailAttachments) .from(emailAttachments)
.innerJoin(attachments, eq(emailAttachments.attachmentId, attachments.id)) .innerJoin(attachments, eq(emailAttachments.attachmentId, attachments.id))
.where(eq(emailAttachments.emailId, emailId)); .where(eq(emailAttachments.emailId, emailId));
try { try {
for (const attachment of emailAttachmentsResult) { for (const attachment of emailAttachmentsResult) {
const [refCount] = await db const [refCount] = await db
.select({ count: count(emailAttachments.emailId) }) .select({ count: count(emailAttachments.emailId) })
.from(emailAttachments) .from(emailAttachments)
.where(eq(emailAttachments.attachmentId, attachment.attachmentId)); .where(eq(emailAttachments.attachmentId, attachment.attachmentId));
if (refCount.count === 1) { if (refCount.count === 1) {
await storage.delete(attachment.storagePath); await storage.delete(attachment.storagePath);
await db await db
.delete(emailAttachments) .delete(emailAttachments)
.where( .where(
and( and(
eq(emailAttachments.emailId, emailId), eq(emailAttachments.emailId, emailId),
eq(emailAttachments.attachmentId, attachment.attachmentId) eq(emailAttachments.attachmentId, attachment.attachmentId)
) )
); );
await db await db
.delete(attachments) .delete(attachments)
.where(eq(attachments.id, attachment.attachmentId)); .where(eq(attachments.id, attachment.attachmentId));
} else { } else {
await db await db
.delete(emailAttachments) .delete(emailAttachments)
.where( .where(
and( and(
eq(emailAttachments.emailId, emailId), eq(emailAttachments.emailId, emailId),
eq(emailAttachments.attachmentId, attachment.attachmentId) eq(emailAttachments.attachmentId, attachment.attachmentId)
) )
); );
} }
} }
} catch { } catch {
throw new Error('Failed to delete email attachments'); throw new Error('Failed to delete email attachments');
} }
} }
// Delete the email file from storage // Delete the email file from storage
await storage.delete(email.storagePath); await storage.delete(email.storagePath);
const searchService = new SearchService(); const searchService = new SearchService();
await searchService.deleteDocuments('emails', [emailId]); await searchService.deleteDocuments('emails', [emailId]);
await db.delete(archivedEmails).where(eq(archivedEmails.id, emailId)); await db.delete(archivedEmails).where(eq(archivedEmails.id, emailId));
} }
} }

110
packages/backend/src/services/AuthService.ts
View File

@@ -7,72 +7,72 @@ import * as schema from '../database/schema';
import { eq } from 'drizzle-orm'; import { eq } from 'drizzle-orm';
export class AuthService { export class AuthService {
#userService: UserService; #userService: UserService;
#jwtSecret: Uint8Array; #jwtSecret: Uint8Array;
#jwtExpiresIn: string; #jwtExpiresIn: string;
constructor(userService: UserService, jwtSecret: string, jwtExpiresIn: string) { constructor(userService: UserService, jwtSecret: string, jwtExpiresIn: string) {
this.#userService = userService; this.#userService = userService;
this.#jwtSecret = new TextEncoder().encode(jwtSecret); this.#jwtSecret = new TextEncoder().encode(jwtSecret);
this.#jwtExpiresIn = jwtExpiresIn; this.#jwtExpiresIn = jwtExpiresIn;
} }
public async verifyPassword(password: string, hash: string): Promise<boolean> { public async verifyPassword(password: string, hash: string): Promise<boolean> {
return compare(password, hash); return compare(password, hash);
} }
async #generateAccessToken(payload: AuthTokenPayload): Promise<string> { async #generateAccessToken(payload: AuthTokenPayload): Promise<string> {
if (!payload.sub) { if (!payload.sub) {
throw new Error('JWT payload must have a subject (sub) claim.'); throw new Error('JWT payload must have a subject (sub) claim.');
} }
return new SignJWT(payload) return new SignJWT(payload)
.setProtectedHeader({ alg: 'HS256' }) .setProtectedHeader({ alg: 'HS256' })
.setIssuedAt() .setIssuedAt()
.setSubject(payload.sub) .setSubject(payload.sub)
.setExpirationTime(this.#jwtExpiresIn) .setExpirationTime(this.#jwtExpiresIn)
.sign(this.#jwtSecret); .sign(this.#jwtSecret);
} }
public async login(email: string, password: string): Promise<LoginResponse | null> { public async login(email: string, password: string): Promise<LoginResponse | null> {
const user = await this.#userService.findByEmail(email); const user = await this.#userService.findByEmail(email);
if (!user || !user.password) { if (!user || !user.password) {
return null; // User not found or password not set return null; // User not found or password not set
} }
const isPasswordValid = await this.verifyPassword(password, user.password); const isPasswordValid = await this.verifyPassword(password, user.password);
if (!isPasswordValid) { if (!isPasswordValid) {
return null; // Invalid password return null; // Invalid password
} }
const userRoles = await db.query.userRoles.findMany({ const userRoles = await db.query.userRoles.findMany({
where: eq(schema.userRoles.userId, user.id), where: eq(schema.userRoles.userId, user.id),
with: { with: {
role: true role: true,
} },
}); });
const roles = userRoles.map(ur => ur.role.name); const roles = userRoles.map((ur) => ur.role.name);
const { password: _, ...userWithoutPassword } = user; const { password: _, ...userWithoutPassword } = user;
const accessToken = await this.#generateAccessToken({ const accessToken = await this.#generateAccessToken({
sub: user.id, sub: user.id,
email: user.email, email: user.email,
roles: roles, roles: roles,
}); });
return { accessToken, user: userWithoutPassword }; return { accessToken, user: userWithoutPassword };
} }
public async verifyToken(token: string): Promise<AuthTokenPayload | null> { public async verifyToken(token: string): Promise<AuthTokenPayload | null> {
try { try {
const { payload } = await jwtVerify<AuthTokenPayload>(token, this.#jwtSecret); const { payload } = await jwtVerify<AuthTokenPayload>(token, this.#jwtSecret);
return payload; return payload;
} catch (error) { } catch (error) {
// Token is invalid or expired // Token is invalid or expired
return null; return null;
} }
} }
} }

91
packages/backend/src/services/CryptoService.ts
View File

@@ -8,63 +8,66 @@ const TAG_LENGTH = 16;
const ENCRYPTION_KEY = config.app.encryptionKey; const ENCRYPTION_KEY = config.app.encryptionKey;
if (!ENCRYPTION_KEY) { if (!ENCRYPTION_KEY) {
throw new Error('ENCRYPTION_KEY is not set in environment variables.'); throw new Error('ENCRYPTION_KEY is not set in environment variables.');
} }
// Derive a key from the master encryption key and a salt // Derive a key from the master encryption key and a salt
const getKey = (salt: Buffer): Buffer => { const getKey = (salt: Buffer): Buffer => {
return scryptSync(ENCRYPTION_KEY, salt, 32); return scryptSync(ENCRYPTION_KEY, salt, 32);
}; };
export class CryptoService { export class CryptoService {
public static encrypt(value: string): string { public static encrypt(value: string): string {
const salt = randomBytes(SALT_LENGTH); const salt = randomBytes(SALT_LENGTH);
const key = getKey(salt); const key = getKey(salt);
const iv = randomBytes(IV_LENGTH); const iv = randomBytes(IV_LENGTH);
const cipher = createCipheriv(ALGORITHM, key, iv); const cipher = createCipheriv(ALGORITHM, key, iv);
const encrypted = Buffer.concat([cipher.update(value, 'utf8'), cipher.final()]); const encrypted = Buffer.concat([cipher.update(value, 'utf8'), cipher.final()]);
const tag = cipher.getAuthTag(); const tag = cipher.getAuthTag();
return Buffer.concat([salt, iv, tag, encrypted]).toString('hex'); return Buffer.concat([salt, iv, tag, encrypted]).toString('hex');
} }
public static decrypt(encrypted: string): string | null { public static decrypt(encrypted: string): string | null {
try { try {
const data = Buffer.from(encrypted, 'hex'); const data = Buffer.from(encrypted, 'hex');
const salt = data.subarray(0, SALT_LENGTH); const salt = data.subarray(0, SALT_LENGTH);
const iv = data.subarray(SALT_LENGTH, SALT_LENGTH + IV_LENGTH); const iv = data.subarray(SALT_LENGTH, SALT_LENGTH + IV_LENGTH);
const tag = data.subarray(SALT_LENGTH + IV_LENGTH, SALT_LENGTH + IV_LENGTH + TAG_LENGTH); const tag = data.subarray(
const encryptedValue = data.subarray(SALT_LENGTH + IV_LENGTH + TAG_LENGTH); SALT_LENGTH + IV_LENGTH,
SALT_LENGTH + IV_LENGTH + TAG_LENGTH
);
const encryptedValue = data.subarray(SALT_LENGTH + IV_LENGTH + TAG_LENGTH);
const key = getKey(salt); const key = getKey(salt);
const decipher = createDecipheriv(ALGORITHM, key, iv); const decipher = createDecipheriv(ALGORITHM, key, iv);
decipher.setAuthTag(tag); decipher.setAuthTag(tag);
const decrypted = Buffer.concat([decipher.update(encryptedValue), decipher.final()]); const decrypted = Buffer.concat([decipher.update(encryptedValue), decipher.final()]);
return decrypted.toString('utf8'); return decrypted.toString('utf8');
} catch (error) { } catch (error) {
console.error('Decryption failed:', error); console.error('Decryption failed:', error);
return null; return null;
} }
} }
public static encryptObject<T extends object>(obj: T): string { public static encryptObject<T extends object>(obj: T): string {
const jsonString = JSON.stringify(obj); const jsonString = JSON.stringify(obj);
return this.encrypt(jsonString); return this.encrypt(jsonString);
} }
public static decryptObject<T extends object>(encrypted: string): T | null { public static decryptObject<T extends object>(encrypted: string): T | null {
const decryptedString = this.decrypt(encrypted); const decryptedString = this.decrypt(encrypted);
if (!decryptedString) { if (!decryptedString) {
return null; return null;
} }
try { try {
return JSON.parse(decryptedString) as T; return JSON.parse(decryptedString) as T;
} catch (error) { } catch (error) {
console.error('Failed to parse decrypted JSON:', error); console.error('Failed to parse decrypted JSON:', error);
return null; return null;
} }
} }
} }

132
packages/backend/src/services/DashboardService.ts
View File

@@ -6,84 +6,84 @@ import { DatabaseService } from './DatabaseService';
import { SearchService } from './SearchService'; import { SearchService } from './SearchService';
class DashboardService { class DashboardService {
#db; #db;
#searchService; #searchService;
constructor(databaseService: DatabaseService, searchService: SearchService) { constructor(databaseService: DatabaseService, searchService: SearchService) {
this.#db = databaseService.db; this.#db = databaseService.db;
this.#searchService = searchService; this.#searchService = searchService;
} }
public async getStats() { public async getStats() {
const totalEmailsArchived = await this.#db.select({ count: count() }).from(archivedEmails); const totalEmailsArchived = await this.#db.select({ count: count() }).from(archivedEmails);
const totalStorageUsed = await this.#db const totalStorageUsed = await this.#db
.select({ sum: sql<number>`sum(${archivedEmails.sizeBytes})` }) .select({ sum: sql<number>`sum(${archivedEmails.sizeBytes})` })
.from(archivedEmails); .from(archivedEmails);
const sevenDaysAgo = new Date(); const sevenDaysAgo = new Date();
sevenDaysAgo.setDate(sevenDaysAgo.getDate() - 7); sevenDaysAgo.setDate(sevenDaysAgo.getDate() - 7);
const failedIngestionsLast7Days = await this.#db const failedIngestionsLast7Days = await this.#db
.select({ count: count() }) .select({ count: count() })
.from(ingestionSources) .from(ingestionSources)
.where( .where(
and( and(
eq(ingestionSources.status, 'error'), eq(ingestionSources.status, 'error'),
gte(ingestionSources.updatedAt, sevenDaysAgo) gte(ingestionSources.updatedAt, sevenDaysAgo)
) )
); );
return { return {
totalEmailsArchived: totalEmailsArchived[0].count, totalEmailsArchived: totalEmailsArchived[0].count,
totalStorageUsed: totalStorageUsed[0].sum || 0, totalStorageUsed: totalStorageUsed[0].sum || 0,
failedIngestionsLast7Days: failedIngestionsLast7Days[0].count failedIngestionsLast7Days: failedIngestionsLast7Days[0].count,
}; };
} }
public async getIngestionHistory() { public async getIngestionHistory() {
const thirtyDaysAgo = new Date(); const thirtyDaysAgo = new Date();
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30); thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);
const history = await this.#db const history = await this.#db
.select({ .select({
date: sql<string>`date_trunc('day', ${archivedEmails.archivedAt})`, date: sql<string>`date_trunc('day', ${archivedEmails.archivedAt})`,
count: count() count: count(),
}) })
.from(archivedEmails) .from(archivedEmails)
.where(gte(archivedEmails.archivedAt, thirtyDaysAgo)) .where(gte(archivedEmails.archivedAt, thirtyDaysAgo))
.groupBy(sql`date_trunc('day', ${archivedEmails.archivedAt})`) .groupBy(sql`date_trunc('day', ${archivedEmails.archivedAt})`)
.orderBy(sql`date_trunc('day', ${archivedEmails.archivedAt})`); .orderBy(sql`date_trunc('day', ${archivedEmails.archivedAt})`);
return { history }; return { history };
} }
public async getIngestionSources() { public async getIngestionSources() {
const sources = await this.#db const sources = await this.#db
.select({ .select({
id: ingestionSources.id, id: ingestionSources.id,
name: ingestionSources.name, name: ingestionSources.name,
provider: ingestionSources.provider, provider: ingestionSources.provider,
status: ingestionSources.status, status: ingestionSources.status,
storageUsed: sql<number>`sum(${archivedEmails.sizeBytes})`.mapWith(Number) storageUsed: sql<number>`sum(${archivedEmails.sizeBytes})`.mapWith(Number),
}) })
.from(ingestionSources) .from(ingestionSources)
.leftJoin(archivedEmails, eq(ingestionSources.id, archivedEmails.ingestionSourceId)) .leftJoin(archivedEmails, eq(ingestionSources.id, archivedEmails.ingestionSourceId))
.groupBy(ingestionSources.id); .groupBy(ingestionSources.id);
return sources; return sources;
} }
public async getRecentSyncs() { public async getRecentSyncs() {
// This is a placeholder as we don't have a sync job table yet. // This is a placeholder as we don't have a sync job table yet.
return Promise.resolve([]); return Promise.resolve([]);
} }
public async getIndexedInsights(): Promise<IndexedInsights> { public async getIndexedInsights(): Promise<IndexedInsights> {
const topSenders = await this.#searchService.getTopSenders(10); const topSenders = await this.#searchService.getTopSenders(10);
return { return {
topSenders topSenders,
}; };
} }
} }
export const dashboardService = new DashboardService(new DatabaseService(), new SearchService()); export const dashboardService = new DashboardService(new DatabaseService(), new SearchService());

3
packages/backend/src/services/DatabaseService.ts
View File

@@ -1,6 +1,5 @@
import { db } from '../database'; import { db } from '../database';
export class DatabaseService { export class DatabaseService {
public db = db; public db = db;
} }

67
packages/backend/src/services/EmailProviderFactory.ts
View File

@@ -1,13 +1,13 @@
import type { import type {
IngestionSource, IngestionSource,
GoogleWorkspaceCredentials, GoogleWorkspaceCredentials,
Microsoft365Credentials, Microsoft365Credentials,
GenericImapCredentials, GenericImapCredentials,
PSTImportCredentials, PSTImportCredentials,
EMLImportCredentials, EMLImportCredentials,
EmailObject, EmailObject,
SyncState, SyncState,
MailboxUser MailboxUser,
} from '@open-archiver/types'; } from '@open-archiver/types';
import { GoogleWorkspaceConnector } from './ingestion-connectors/GoogleWorkspaceConnector'; import { GoogleWorkspaceConnector } from './ingestion-connectors/GoogleWorkspaceConnector';
import { MicrosoftConnector } from './ingestion-connectors/MicrosoftConnector'; import { MicrosoftConnector } from './ingestion-connectors/MicrosoftConnector';
@@ -17,31 +17,34 @@ import { EMLConnector } from './ingestion-connectors/EMLConnector';
// Define a common interface for all connectors // Define a common interface for all connectors
export interface IEmailConnector { export interface IEmailConnector {
testConnection(): Promise<boolean>; testConnection(): Promise<boolean>;
fetchEmails(userEmail: string, syncState?: SyncState | null): AsyncGenerator<EmailObject | null>; fetchEmails(
getUpdatedSyncState(userEmail?: string): SyncState; userEmail: string,
listAllUsers(): AsyncGenerator<MailboxUser>; syncState?: SyncState | null
returnImapUserEmail?(): string; ): AsyncGenerator<EmailObject | null>;
getUpdatedSyncState(userEmail?: string): SyncState;
listAllUsers(): AsyncGenerator<MailboxUser>;
returnImapUserEmail?(): string;
} }
export class EmailProviderFactory { export class EmailProviderFactory {
static createConnector(source: IngestionSource): IEmailConnector { static createConnector(source: IngestionSource): IEmailConnector {
// Credentials are now decrypted by the IngestionService before being passed around // Credentials are now decrypted by the IngestionService before being passed around
const credentials = source.credentials; const credentials = source.credentials;
switch (source.provider) { switch (source.provider) {
case 'google_workspace': case 'google_workspace':
return new GoogleWorkspaceConnector(credentials as GoogleWorkspaceCredentials); return new GoogleWorkspaceConnector(credentials as GoogleWorkspaceCredentials);
case 'microsoft_365': case 'microsoft_365':
return new MicrosoftConnector(credentials as Microsoft365Credentials); return new MicrosoftConnector(credentials as Microsoft365Credentials);
case 'generic_imap': case 'generic_imap':
return new ImapConnector(credentials as GenericImapCredentials); return new ImapConnector(credentials as GenericImapCredentials);
case 'pst_import': case 'pst_import':
return new PSTConnector(credentials as PSTImportCredentials); return new PSTConnector(credentials as PSTImportCredentials);
case 'eml_import': case 'eml_import':
return new EMLConnector(credentials as EMLImportCredentials); return new EMLConnector(credentials as EMLImportCredentials);
default: default:
throw new Error(`Unsupported provider: ${source.provider}`); throw new Error(`Unsupported provider: ${source.provider}`);
} }
} }
} }

28
packages/backend/src/services/IamService.ts
View File

@@ -4,21 +4,21 @@ import type { Role, PolicyStatement } from '@open-archiver/types';
import { eq } from 'drizzle-orm'; import { eq } from 'drizzle-orm';
export class IamService { export class IamService {
public async getRoles(): Promise<Role[]> { public async getRoles(): Promise<Role[]> {
return db.select().from(roles); return db.select().from(roles);
} }
public async getRoleById(id: string): Promise<Role | undefined> { public async getRoleById(id: string): Promise<Role | undefined> {
const [role] = await db.select().from(roles).where(eq(roles.id, id)); const [role] = await db.select().from(roles).where(eq(roles.id, id));
return role; return role;
} }
public async createRole(name: string, policy: PolicyStatement[]): Promise<Role> { public async createRole(name: string, policy: PolicyStatement[]): Promise<Role> {
const [role] = await db.insert(roles).values({ name, policies: policy }).returning(); const [role] = await db.insert(roles).values({ name, policies: policy }).returning();
return role; return role;
} }
public async deleteRole(id: string): Promise<void> { public async deleteRole(id: string): Promise<void> {
await db.delete(roles).where(eq(roles.id, id)); await db.delete(roles).where(eq(roles.id, id));
} }
} }

339
packages/backend/src/services/IndexingService.ts
View File

@@ -9,192 +9,191 @@ import { streamToBuffer } from '../helpers/streamToBuffer';
import { simpleParser } from 'mailparser'; import { simpleParser } from 'mailparser';
interface DbRecipients { interface DbRecipients {
to: { name: string; address: string; }[]; to: { name: string; address: string }[];
cc: { name: string; address: string; }[]; cc: { name: string; address: string }[];
bcc: { name: string; address: string; }[]; bcc: { name: string; address: string }[];
} }
type AttachmentsType = { type AttachmentsType = {
filename: string, filename: string;
buffer: Buffer, buffer: Buffer;
mimeType: string; mimeType: string;
}[]; }[];
export class IndexingService { export class IndexingService {
private dbService: DatabaseService; private dbService: DatabaseService;
private searchService: SearchService; private searchService: SearchService;
private storageService: StorageService; private storageService: StorageService;
/** /**
* Initializes the service with its dependencies. * Initializes the service with its dependencies.
*/ */
constructor( constructor(
dbService: DatabaseService, dbService: DatabaseService,
searchService: SearchService, searchService: SearchService,
storageService: StorageService, storageService: StorageService
) { ) {
this.dbService = dbService; this.dbService = dbService;
this.searchService = searchService; this.searchService = searchService;
this.storageService = storageService; this.storageService = storageService;
} }
/** /**
* Fetches an email by its ID from the database, creates a search document, and indexes it. * Fetches an email by its ID from the database, creates a search document, and indexes it.
*/ */
public async indexEmailById(emailId: string): Promise<void> { public async indexEmailById(emailId: string): Promise<void> {
const email = await this.dbService.db.query.archivedEmails.findFirst({ const email = await this.dbService.db.query.archivedEmails.findFirst({
where: eq(archivedEmails.id, emailId), where: eq(archivedEmails.id, emailId),
}); });
if (!email) { if (!email) {
throw new Error(`Email with ID ${emailId} not found for indexing.`); throw new Error(`Email with ID ${emailId} not found for indexing.`);
} }
let emailAttachmentsResult: Attachment[] = []; let emailAttachmentsResult: Attachment[] = [];
if (email.hasAttachments) { if (email.hasAttachments) {
emailAttachmentsResult = await this.dbService.db emailAttachmentsResult = await this.dbService.db
.select({ .select({
id: attachments.id, id: attachments.id,
filename: attachments.filename, filename: attachments.filename,
mimeType: attachments.mimeType, mimeType: attachments.mimeType,
sizeBytes: attachments.sizeBytes, sizeBytes: attachments.sizeBytes,
contentHashSha256: attachments.contentHashSha256, contentHashSha256: attachments.contentHashSha256,
storagePath: attachments.storagePath, storagePath: attachments.storagePath,
}) })
.from(emailAttachments) .from(emailAttachments)
.innerJoin(attachments, eq(emailAttachments.attachmentId, attachments.id)) .innerJoin(attachments, eq(emailAttachments.attachmentId, attachments.id))
.where(eq(emailAttachments.emailId, emailId)); .where(eq(emailAttachments.emailId, emailId));
} }
const document = await this.createEmailDocument(email, emailAttachmentsResult); const document = await this.createEmailDocument(email, emailAttachmentsResult);
await this.searchService.addDocuments('emails', [document], 'id'); await this.searchService.addDocuments('emails', [document], 'id');
} }
/** /**
* Indexes an email object directly, creates a search document, and indexes it. * Indexes an email object directly, creates a search document, and indexes it.
*/ */
public async indexByEmail(email: EmailObject, ingestionSourceId: string, archivedEmailId: string): Promise<void> { public async indexByEmail(
const attachments: AttachmentsType = []; email: EmailObject,
if (email.attachments && email.attachments.length > 0) { ingestionSourceId: string,
for (const attachment of email.attachments) { archivedEmailId: string
attachments.push({ ): Promise<void> {
buffer: attachment.content, const attachments: AttachmentsType = [];
filename: attachment.filename, if (email.attachments && email.attachments.length > 0) {
mimeType: attachment.contentType for (const attachment of email.attachments) {
}); attachments.push({
} buffer: attachment.content,
} filename: attachment.filename,
const document = await this.createEmailDocumentFromRaw(email, attachments, ingestionSourceId, archivedEmailId); mimeType: attachment.contentType,
await this.searchService.addDocuments('emails', [document], 'id'); });
} }
}
const document = await this.createEmailDocumentFromRaw(
email,
attachments,
ingestionSourceId,
archivedEmailId
);
await this.searchService.addDocuments('emails', [document], 'id');
}
/** /**
* Creates a search document from a raw email object and its attachments. * Creates a search document from a raw email object and its attachments.
*/ */
private async createEmailDocumentFromRaw( private async createEmailDocumentFromRaw(
email: EmailObject, email: EmailObject,
attachments: AttachmentsType, attachments: AttachmentsType,
ingestionSourceId: string, ingestionSourceId: string,
archivedEmailId: string archivedEmailId: string
): Promise<EmailDocument> { ): Promise<EmailDocument> {
const extractedAttachments = []; const extractedAttachments = [];
for (const attachment of attachments) { for (const attachment of attachments) {
try { try {
const textContent = await extractText( const textContent = await extractText(attachment.buffer, attachment.mimeType || '');
attachment.buffer, extractedAttachments.push({
attachment.mimeType || '' filename: attachment.filename,
); content: textContent,
extractedAttachments.push({ });
filename: attachment.filename, } catch (error) {
content: textContent, console.error(
}); `Failed to extract text from attachment: ${attachment.filename}`,
} catch (error) { error
console.error( );
`Failed to extract text from attachment: ${attachment.filename}`, // skip attachment or fail the job
error }
); }
// skip attachment or fail the job return {
} id: archivedEmailId,
} from: email.from[0]?.address,
return { to: email.to.map((i: EmailAddress) => i.address) || [],
id: archivedEmailId, cc: email.cc?.map((i: EmailAddress) => i.address) || [],
from: email.from[0]?.address, bcc: email.bcc?.map((i: EmailAddress) => i.address) || [],
to: email.to.map((i: EmailAddress) => i.address) || [], subject: email.subject || '',
cc: email.cc?.map((i: EmailAddress) => i.address) || [], body: email.body || email.html || '',
bcc: email.bcc?.map((i: EmailAddress) => i.address) || [], attachments: extractedAttachments,
subject: email.subject || '', timestamp: new Date(email.receivedAt).getTime(),
body: email.body || email.html || '', ingestionSourceId: ingestionSourceId,
attachments: extractedAttachments, };
timestamp: new Date(email.receivedAt).getTime(), }
ingestionSourceId: ingestionSourceId
};
}
/**
* Creates a search document from a database email record and its attachments.
*/
private async createEmailDocument(
email: typeof archivedEmails.$inferSelect,
attachments: Attachment[]
): Promise<EmailDocument> {
const attachmentContents = await this.extractAttachmentContents(attachments);
/** const emailBodyStream = await this.storageService.get(email.storagePath);
* Creates a search document from a database email record and its attachments. const emailBodyBuffer = await streamToBuffer(emailBodyStream);
*/ const parsedEmail = await simpleParser(emailBodyBuffer);
private async createEmailDocument( const emailBodyText =
email: typeof archivedEmails.$inferSelect, parsedEmail.text ||
attachments: Attachment[] parsedEmail.html ||
): Promise<EmailDocument> { (await extractText(emailBodyBuffer, 'text/plain')) ||
const attachmentContents = await this.extractAttachmentContents(attachments); '';
const emailBodyStream = await this.storageService.get(email.storagePath); const recipients = email.recipients as DbRecipients;
const emailBodyBuffer = await streamToBuffer(emailBodyStream);
const parsedEmail = await simpleParser(emailBodyBuffer);
const emailBodyText =
parsedEmail.text ||
parsedEmail.html ||
(await extractText(emailBodyBuffer, 'text/plain')) ||
'';
const recipients = email.recipients as DbRecipients; return {
id: email.id,
return { from: email.senderEmail,
id: email.id, to: recipients.to?.map((r) => r.address) || [],
from: email.senderEmail, cc: recipients.cc?.map((r) => r.address) || [],
to: recipients.to?.map((r) => r.address) || [], bcc: recipients.bcc?.map((r) => r.address) || [],
cc: recipients.cc?.map((r) => r.address) || [], subject: email.subject || '',
bcc: recipients.bcc?.map((r) => r.address) || [], body: emailBodyText,
subject: email.subject || '', attachments: attachmentContents,
body: emailBodyText, timestamp: new Date(email.sentAt).getTime(),
attachments: attachmentContents, ingestionSourceId: email.ingestionSourceId,
timestamp: new Date(email.sentAt).getTime(), };
ingestionSourceId: email.ingestionSourceId }
};
}
/**
* Extracts text content from a list of attachments.
*/
private async extractAttachmentContents(
attachments: Attachment[]
): Promise<{ filename: string; content: string; }[]> {
const extractedAttachments = [];
for (const attachment of attachments) {
try {
const fileStream = await this.storageService.get(
attachment.storagePath
);
const fileBuffer = await streamToBuffer(fileStream);
const textContent = await extractText(
fileBuffer,
attachment.mimeType || ''
);
extractedAttachments.push({
filename: attachment.filename,
content: textContent,
});
} catch (error) {
console.error(
`Failed to extract text from attachment: ${attachment.filename}`,
error
);
// skip attachment or fail the job
}
}
return extractedAttachments;
}
/**
* Extracts text content from a list of attachments.
*/
private async extractAttachmentContents(
attachments: Attachment[]
): Promise<{ filename: string; content: string }[]> {
const extractedAttachments = [];
for (const attachment of attachments) {
try {
const fileStream = await this.storageService.get(attachment.storagePath);
const fileBuffer = await streamToBuffer(fileStream);
const textContent = await extractText(fileBuffer, attachment.mimeType || '');
extractedAttachments.push({
filename: attachment.filename,
content: textContent,
});
} catch (error) {
console.error(
`Failed to extract text from attachment: ${attachment.filename}`,
error
);
// skip attachment or fail the job
}
}
return extractedAttachments;
}
} }

658
packages/backend/src/services/IngestionService.ts
View File

@@ -1,11 +1,11 @@
import { db } from '../database'; import { db } from '../database';
import { ingestionSources } from '../database/schema'; import { ingestionSources } from '../database/schema';
import type { import type {
CreateIngestionSourceDto, CreateIngestionSourceDto,
UpdateIngestionSourceDto, UpdateIngestionSourceDto,
IngestionSource, IngestionSource,
IngestionCredentials, IngestionCredentials,
IngestionProvider IngestionProvider,
} from '@open-archiver/types'; } from '@open-archiver/types';
import { and, desc, eq } from 'drizzle-orm'; import { and, desc, eq } from 'drizzle-orm';
import { CryptoService } from './CryptoService'; import { CryptoService } from './CryptoService';
@@ -14,7 +14,11 @@ import { ingestionQueue } from '../jobs/queues';
import type { JobType } from 'bullmq'; import type { JobType } from 'bullmq';
import { StorageService } from './StorageService'; import { StorageService } from './StorageService';
import type { IInitialImportJob, EmailObject } from '@open-archiver/types'; import type { IInitialImportJob, EmailObject } from '@open-archiver/types';
import { archivedEmails, attachments as attachmentsSchema, emailAttachments } from '../database/schema'; import {
archivedEmails,
attachments as attachmentsSchema,
emailAttachments,
} from '../database/schema';
import { createHash } from 'crypto'; import { createHash } from 'crypto';
import { logger } from '../config/logger'; import { logger } from '../config/logger';
import { IndexingService } from './IndexingService'; import { IndexingService } from './IndexingService';
@@ -22,352 +26,386 @@ import { SearchService } from './SearchService';
import { DatabaseService } from './DatabaseService'; import { DatabaseService } from './DatabaseService';
import { config } from '../config/index'; import { config } from '../config/index';
export class IngestionService { export class IngestionService {
private static decryptSource(source: typeof ingestionSources.$inferSelect): IngestionSource | null { private static decryptSource(
const decryptedCredentials = CryptoService.decryptObject<IngestionCredentials>( source: typeof ingestionSources.$inferSelect
source.credentials as string ): IngestionSource | null {
); const decryptedCredentials = CryptoService.decryptObject<IngestionCredentials>(
source.credentials as string
);
if (!decryptedCredentials) { if (!decryptedCredentials) {
logger.error({ sourceId: source.id }, 'Failed to decrypt ingestion source credentials.'); logger.error(
return null; { sourceId: source.id },
} 'Failed to decrypt ingestion source credentials.'
);
return null;
}
return { ...source, credentials: decryptedCredentials } as IngestionSource; return { ...source, credentials: decryptedCredentials } as IngestionSource;
} }
public static returnFileBasedIngestions(): IngestionProvider[] { public static returnFileBasedIngestions(): IngestionProvider[] {
return ['pst_import', 'eml_import']; return ['pst_import', 'eml_import'];
} }
public static async create(dto: CreateIngestionSourceDto): Promise<IngestionSource> { public static async create(dto: CreateIngestionSourceDto): Promise<IngestionSource> {
const { providerConfig, ...rest } = dto; const { providerConfig, ...rest } = dto;
const encryptedCredentials = CryptoService.encryptObject(providerConfig); const encryptedCredentials = CryptoService.encryptObject(providerConfig);
const valuesToInsert = { const valuesToInsert = {
...rest, ...rest,
status: 'pending_auth' as const, status: 'pending_auth' as const,
credentials: encryptedCredentials credentials: encryptedCredentials,
}; };
const [newSource] = await db.insert(ingestionSources).values(valuesToInsert).returning(); const [newSource] = await db.insert(ingestionSources).values(valuesToInsert).returning();
const decryptedSource = this.decryptSource(newSource); const decryptedSource = this.decryptSource(newSource);
if (!decryptedSource) { if (!decryptedSource) {
await this.delete(newSource.id); await this.delete(newSource.id);
throw new Error('Failed to process newly created ingestion source due to a decryption error.'); throw new Error(
} 'Failed to process newly created ingestion source due to a decryption error.'
const connector = EmailProviderFactory.createConnector(decryptedSource); );
}
const connector = EmailProviderFactory.createConnector(decryptedSource);
try { try {
await connector.testConnection(); await connector.testConnection();
// If connection succeeds, update status to auth_success, which triggers the initial import. // If connection succeeds, update status to auth_success, which triggers the initial import.
return await this.update(decryptedSource.id, { status: 'auth_success' }); return await this.update(decryptedSource.id, { status: 'auth_success' });
} catch (error) { } catch (error) {
// If connection fails, delete the newly created source and throw the error. // If connection fails, delete the newly created source and throw the error.
await this.delete(decryptedSource.id); await this.delete(decryptedSource.id);
throw error; throw error;
} }
} }
public static async findAll(): Promise<IngestionSource[]> { public static async findAll(): Promise<IngestionSource[]> {
const sources = await db.select().from(ingestionSources).orderBy(desc(ingestionSources.createdAt)); const sources = await db
return sources.flatMap(source => { .select()
const decrypted = this.decryptSource(source); .from(ingestionSources)
return decrypted ? [decrypted] : []; .orderBy(desc(ingestionSources.createdAt));
}); return sources.flatMap((source) => {
} const decrypted = this.decryptSource(source);
return decrypted ? [decrypted] : [];
});
}
public static async findById(id: string): Promise<IngestionSource> { public static async findById(id: string): Promise<IngestionSource> {
const [source] = await db.select().from(ingestionSources).where(eq(ingestionSources.id, id)); const [source] = await db
if (!source) { .select()
throw new Error('Ingestion source not found'); .from(ingestionSources)
} .where(eq(ingestionSources.id, id));
const decryptedSource = this.decryptSource(source); if (!source) {
if (!decryptedSource) { throw new Error('Ingestion source not found');
throw new Error('Failed to decrypt ingestion source credentials.'); }
} const decryptedSource = this.decryptSource(source);
return decryptedSource; if (!decryptedSource) {
} throw new Error('Failed to decrypt ingestion source credentials.');
}
return decryptedSource;
}
public static async update( public static async update(
id: string, id: string,
dto: UpdateIngestionSourceDto dto: UpdateIngestionSourceDto
): Promise<IngestionSource> { ): Promise<IngestionSource> {
const { providerConfig, ...rest } = dto; const { providerConfig, ...rest } = dto;
const valuesToUpdate: Partial<typeof ingestionSources.$inferInsert> = { ...rest }; const valuesToUpdate: Partial<typeof ingestionSources.$inferInsert> = { ...rest };
// Get the original source to compare the status later // Get the original source to compare the status later
const originalSource = await this.findById(id); const originalSource = await this.findById(id);
if (providerConfig) { if (providerConfig) {
// Encrypt the new credentials before updating // Encrypt the new credentials before updating
valuesToUpdate.credentials = CryptoService.encryptObject(providerConfig); valuesToUpdate.credentials = CryptoService.encryptObject(providerConfig);
} }
const [updatedSource] = await db const [updatedSource] = await db
.update(ingestionSources) .update(ingestionSources)
.set(valuesToUpdate) .set(valuesToUpdate)
.where(eq(ingestionSources.id, id)) .where(eq(ingestionSources.id, id))
.returning(); .returning();
if (!updatedSource) { if (!updatedSource) {
throw new Error('Ingestion source not found'); throw new Error('Ingestion source not found');
} }
const decryptedSource = this.decryptSource(updatedSource); const decryptedSource = this.decryptSource(updatedSource);
if (!decryptedSource) { if (!decryptedSource) {
throw new Error('Failed to process updated ingestion source due to a decryption error.'); throw new Error(
} 'Failed to process updated ingestion source due to a decryption error.'
);
}
// If the status has changed to auth_success, trigger the initial import // If the status has changed to auth_success, trigger the initial import
if ( if (originalSource.status !== 'auth_success' && decryptedSource.status === 'auth_success') {
originalSource.status !== 'auth_success' && await this.triggerInitialImport(decryptedSource.id);
decryptedSource.status === 'auth_success' }
) {
await this.triggerInitialImport(decryptedSource.id);
}
return decryptedSource; return decryptedSource;
} }
public static async delete(id: string): Promise<IngestionSource> { public static async delete(id: string): Promise<IngestionSource> {
const source = await this.findById(id); const source = await this.findById(id);
if (!source) { if (!source) {
throw new Error('Ingestion source not found'); throw new Error('Ingestion source not found');
} }
// Delete all emails and attachments from storage // Delete all emails and attachments from storage
const storage = new StorageService(); const storage = new StorageService();
const emailPath = `${config.storage.openArchiverFolderName}/${source.name.replaceAll(' ', '-')}-${source.id}/`; const emailPath = `${config.storage.openArchiverFolderName}/${source.name.replaceAll(' ', '-')}-${source.id}/`;
await storage.delete(emailPath); await storage.delete(emailPath);
if ( if (
(source.credentials.type === 'pst_import' || source.credentials.type === 'eml_import') && (source.credentials.type === 'pst_import' ||
source.credentials.uploadedFilePath && source.credentials.type === 'eml_import') &&
(await storage.exists(source.credentials.uploadedFilePath)) source.credentials.uploadedFilePath &&
) { (await storage.exists(source.credentials.uploadedFilePath))
await storage.delete(source.credentials.uploadedFilePath); ) {
} await storage.delete(source.credentials.uploadedFilePath);
}
// Delete all emails from the database // Delete all emails from the database
// NOTE: This is done by database CASADE, change when CASADE relation no longer exists. // NOTE: This is done by database CASADE, change when CASADE relation no longer exists.
// await db.delete(archivedEmails).where(eq(archivedEmails.ingestionSourceId, id)); // await db.delete(archivedEmails).where(eq(archivedEmails.ingestionSourceId, id));
// Delete all documents from Meilisearch // Delete all documents from Meilisearch
const searchService = new SearchService(); const searchService = new SearchService();
await searchService.deleteDocumentsByFilter('emails', `ingestionSourceId = ${id}`); await searchService.deleteDocumentsByFilter('emails', `ingestionSourceId = ${id}`);
const [deletedSource] = await db const [deletedSource] = await db
.delete(ingestionSources) .delete(ingestionSources)
.where(eq(ingestionSources.id, id)) .where(eq(ingestionSources.id, id))
.returning(); .returning();
const decryptedSource = this.decryptSource(deletedSource); const decryptedSource = this.decryptSource(deletedSource);
if (!decryptedSource) { if (!decryptedSource) {
// Even if decryption fails, we should confirm deletion. // Even if decryption fails, we should confirm deletion.
// We might return a simpler object or just a success message. // We might return a simpler object or just a success message.
// For now, we'll indicate the issue but still confirm deletion happened. // For now, we'll indicate the issue but still confirm deletion happened.
logger.warn({ sourceId: deletedSource.id }, 'Could not decrypt credentials of deleted source, but deletion was successful.'); logger.warn(
return { ...deletedSource, credentials: null } as unknown as IngestionSource; { sourceId: deletedSource.id },
} 'Could not decrypt credentials of deleted source, but deletion was successful.'
return decryptedSource; );
} return { ...deletedSource, credentials: null } as unknown as IngestionSource;
}
return decryptedSource;
}
public static async triggerInitialImport(id: string): Promise<void> { public static async triggerInitialImport(id: string): Promise<void> {
const source = await this.findById(id); const source = await this.findById(id);
await ingestionQueue.add('initial-import', { ingestionSourceId: source.id }); await ingestionQueue.add('initial-import', { ingestionSourceId: source.id });
}
} public static async triggerForceSync(id: string): Promise<void> {
const source = await this.findById(id);
logger.info({ ingestionSourceId: id }, 'Force syncing started.');
if (!source) {
throw new Error('Ingestion source not found');
}
public static async triggerForceSync(id: string): Promise<void> { // Clean up existing jobs for this source to break any stuck flows
const source = await this.findById(id); const jobTypes: JobType[] = ['active', 'waiting', 'failed', 'delayed', 'paused'];
logger.info({ ingestionSourceId: id }, 'Force syncing started.'); const jobs = await ingestionQueue.getJobs(jobTypes);
if (!source) { for (const job of jobs) {
throw new Error('Ingestion source not found'); if (job.data.ingestionSourceId === id) {
} try {
await job.remove();
logger.info(
{ jobId: job.id, ingestionSourceId: id },
'Removed stale job during force sync.'
);
} catch (error) {
logger.error({ err: error, jobId: job.id }, 'Failed to remove stale job.');
}
}
}
// Clean up existing jobs for this source to break any stuck flows // Reset status to 'active'
const jobTypes: JobType[] = ['active', 'waiting', 'failed', 'delayed', 'paused']; await this.update(id, {
const jobs = await ingestionQueue.getJobs(jobTypes); status: 'active',
for (const job of jobs) { lastSyncStatusMessage: 'Force sync triggered by user.',
if (job.data.ingestionSourceId === id) { });
try {
await job.remove();
logger.info({ jobId: job.id, ingestionSourceId: id }, 'Removed stale job during force sync.');
} catch (error) {
logger.error({ err: error, jobId: job.id }, 'Failed to remove stale job.');
}
}
}
// Reset status to 'active' await ingestionQueue.add('continuous-sync', { ingestionSourceId: source.id });
await this.update(id, { status: 'active', lastSyncStatusMessage: 'Force sync triggered by user.' }); }
public async performBulkImport(job: IInitialImportJob): Promise<void> {
const { ingestionSourceId } = job;
const source = await IngestionService.findById(ingestionSourceId);
if (!source) {
throw new Error(`Ingestion source ${ingestionSourceId} not found.`);
}
await ingestionQueue.add('continuous-sync', { ingestionSourceId: source.id }); logger.info(`Starting bulk import for source: ${source.name} (${source.id})`);
} await IngestionService.update(ingestionSourceId, {
status: 'importing',
lastSyncStartedAt: new Date(),
});
public async performBulkImport(job: IInitialImportJob): Promise<void> { const connector = EmailProviderFactory.createConnector(source);
const { ingestionSourceId } = job;
const source = await IngestionService.findById(ingestionSourceId);
if (!source) {
throw new Error(`Ingestion source ${ingestionSourceId} not found.`);
}
logger.info(`Starting bulk import for source: ${source.name} (${source.id})`); try {
await IngestionService.update(ingestionSourceId, { if (connector.listAllUsers) {
status: 'importing', // For multi-mailbox providers, dispatch a job for each user
lastSyncStartedAt: new Date() for await (const user of connector.listAllUsers()) {
}); const userEmail = user.primaryEmail;
if (userEmail) {
await ingestionQueue.add('process-mailbox', {
ingestionSourceId: source.id,
userEmail: userEmail,
});
}
}
} else {
// For single-mailbox providers, dispatch a single job
await ingestionQueue.add('process-mailbox', {
ingestionSourceId: source.id,
userEmail:
source.credentials.type === 'generic_imap'
? source.credentials.username
: 'Default',
});
}
} catch (error) {
logger.error(`Bulk import failed for source: ${source.name} (${source.id})`, error);
await IngestionService.update(ingestionSourceId, {
status: 'error',
lastSyncFinishedAt: new Date(),
lastSyncStatusMessage:
error instanceof Error ? error.message : 'An unknown error occurred.',
});
throw error; // Re-throw to allow BullMQ to handle the job failure
}
}
const connector = EmailProviderFactory.createConnector(source); public async processEmail(
email: EmailObject,
source: IngestionSource,
storage: StorageService,
userEmail: string
): Promise<void> {
try {
// Generate a unique message ID for the email. If the email already has a message-id header, use that.
// Otherwise, generate a new one based on the email's hash, source ID, and email ID.
const messageIdHeader = email.headers.get('message-id');
let messageId: string | undefined;
if (Array.isArray(messageIdHeader)) {
messageId = messageIdHeader[0];
} else if (typeof messageIdHeader === 'string') {
messageId = messageIdHeader;
}
if (!messageId) {
messageId = `generated-${createHash('sha256')
.update(email.eml ?? Buffer.from(email.body, 'utf-8'))
.digest('hex')}-${source.id}-${email.id}`;
}
// Check if an email with the same message ID has already been imported for the current ingestion source. This is to prevent duplicate imports when an email is present in multiple mailboxes (e.g., "Inbox" and "All Mail").
const existingEmail = await db.query.archivedEmails.findFirst({
where: and(
eq(archivedEmails.messageIdHeader, messageId),
eq(archivedEmails.ingestionSourceId, source.id)
),
});
try { if (existingEmail) {
if (connector.listAllUsers) { logger.info(
// For multi-mailbox providers, dispatch a job for each user { messageId, ingestionSourceId: source.id },
for await (const user of connector.listAllUsers()) { 'Skipping duplicate email'
const userEmail = user.primaryEmail; );
if (userEmail) { return;
await ingestionQueue.add('process-mailbox', { }
ingestionSourceId: source.id,
userEmail: userEmail,
});
}
}
} else {
// For single-mailbox providers, dispatch a single job
await ingestionQueue.add('process-mailbox', {
ingestionSourceId: source.id,
userEmail: source.credentials.type === 'generic_imap' ? source.credentials.username : 'Default'
});
}
} catch (error) {
logger.error(`Bulk import failed for source: ${source.name} (${source.id})`, error);
await IngestionService.update(ingestionSourceId, {
status: 'error',
lastSyncFinishedAt: new Date(),
lastSyncStatusMessage: error instanceof Error ? error.message : 'An unknown error occurred.'
});
throw error; // Re-throw to allow BullMQ to handle the job failure
}
}
public async processEmail( const emlBuffer = email.eml ?? Buffer.from(email.body, 'utf-8');
email: EmailObject, const emailHash = createHash('sha256').update(emlBuffer).digest('hex');
source: IngestionSource, const sanitizedPath = email.path ? email.path : '';
storage: StorageService, const emailPath = `${config.storage.openArchiverFolderName}/${source.name.replaceAll(' ', '-')}-${source.id}/emails/${sanitizedPath}${email.id}.eml`;
userEmail: string await storage.put(emailPath, emlBuffer);
): Promise<void> {
try {
// Generate a unique message ID for the email. If the email already has a message-id header, use that.
// Otherwise, generate a new one based on the email's hash, source ID, and email ID.
const messageIdHeader = email.headers.get('message-id');
let messageId: string | undefined;
if (Array.isArray(messageIdHeader)) {
messageId = messageIdHeader[0];
} else if (typeof messageIdHeader === 'string') {
messageId = messageIdHeader;
}
if (!messageId) {
messageId = `generated-${createHash('sha256').update(email.eml ?? Buffer.from(email.body, 'utf-8')).digest('hex')}-${source.id}-${email.id}`;
}
// Check if an email with the same message ID has already been imported for the current ingestion source. This is to prevent duplicate imports when an email is present in multiple mailboxes (e.g., "Inbox" and "All Mail").
const existingEmail = await db.query.archivedEmails.findFirst({
where: and(
eq(archivedEmails.messageIdHeader, messageId),
eq(archivedEmails.ingestionSourceId, source.id)
)
});
if (existingEmail) { const [archivedEmail] = await db
logger.info({ messageId, ingestionSourceId: source.id }, 'Skipping duplicate email'); .insert(archivedEmails)
return; .values({
} ingestionSourceId: source.id,
userEmail,
threadId: email.threadId,
messageIdHeader: messageId,
sentAt: email.receivedAt,
subject: email.subject,
senderName: email.from[0]?.name,
senderEmail: email.from[0]?.address,
recipients: {
to: email.to,
cc: email.cc,
bcc: email.bcc,
},
storagePath: emailPath,
storageHashSha256: emailHash,
sizeBytes: emlBuffer.length,
hasAttachments: email.attachments.length > 0,
path: email.path,
tags: email.tags,
})
.returning();
const emlBuffer = email.eml ?? Buffer.from(email.body, 'utf-8'); if (email.attachments.length > 0) {
const emailHash = createHash('sha256').update(emlBuffer).digest('hex'); for (const attachment of email.attachments) {
const sanitizedPath = email.path ? email.path : ''; const attachmentBuffer = attachment.content;
const emailPath = `${config.storage.openArchiverFolderName}/${source.name.replaceAll(' ', '-')}-${source.id}/emails/${sanitizedPath}${email.id}.eml`; const attachmentHash = createHash('sha256')
await storage.put(emailPath, emlBuffer); .update(attachmentBuffer)
.digest('hex');
const attachmentPath = `${config.storage.openArchiverFolderName}/${source.name.replaceAll(' ', '-')}-${source.id}/attachments/${attachment.filename}`;
await storage.put(attachmentPath, attachmentBuffer);
const [archivedEmail] = await db const [newAttachment] = await db
.insert(archivedEmails) .insert(attachmentsSchema)
.values({ .values({
ingestionSourceId: source.id, filename: attachment.filename,
userEmail, mimeType: attachment.contentType,
threadId: email.threadId, sizeBytes: attachment.size,
messageIdHeader: messageId, contentHashSha256: attachmentHash,
sentAt: email.receivedAt, storagePath: attachmentPath,
subject: email.subject, })
senderName: email.from[0]?.name, .onConflictDoUpdate({
senderEmail: email.from[0]?.address, target: attachmentsSchema.contentHashSha256,
recipients: { set: { filename: attachment.filename },
to: email.to, })
cc: email.cc, .returning();
bcc: email.bcc
},
storagePath: emailPath,
storageHashSha256: emailHash,
sizeBytes: emlBuffer.length,
hasAttachments: email.attachments.length > 0,
path: email.path,
tags: email.tags
})
.returning();
if (email.attachments.length > 0) { await db
for (const attachment of email.attachments) { .insert(emailAttachments)
const attachmentBuffer = attachment.content; .values({
const attachmentHash = createHash('sha256').update(attachmentBuffer).digest('hex'); emailId: archivedEmail.id,
const attachmentPath = `${config.storage.openArchiverFolderName}/${source.name.replaceAll(' ', '-')}-${source.id}/attachments/${attachment.filename}`; attachmentId: newAttachment.id,
await storage.put(attachmentPath, attachmentBuffer); })
.onConflictDoNothing();
const [newAttachment] = await db }
.insert(attachmentsSchema) }
.values({ // adding to indexing queue
filename: attachment.filename, //Instead: index by email (raw email object, ingestion id)
mimeType: attachment.contentType, logger.info({ emailId: archivedEmail.id }, 'Indexing email');
sizeBytes: attachment.size, // await indexingQueue.add('index-email', {
contentHashSha256: attachmentHash, // emailId: archivedEmail.id,
storagePath: attachmentPath // });
}) const searchService = new SearchService();
.onConflictDoUpdate({ const storageService = new StorageService();
target: attachmentsSchema.contentHashSha256, const databaseService = new DatabaseService();
set: { filename: attachment.filename } const indexingService = new IndexingService(
}) databaseService,
.returning(); searchService,
storageService
await db );
.insert(emailAttachments) await indexingService.indexByEmail(email, source.id, archivedEmail.id);
.values({ } catch (error) {
emailId: archivedEmail.id, logger.error({
attachmentId: newAttachment.id message: `Failed to process email ${email.id} for source ${source.id}`,
}) error,
.onConflictDoNothing(); emailId: email.id,
} ingestionSourceId: source.id,
} });
// adding to indexing queue }
//Instead: index by email (raw email object, ingestion id) }
logger.info({ emailId: archivedEmail.id }, 'Indexing email');
// await indexingQueue.add('index-email', {
// emailId: archivedEmail.id,
// });
const searchService = new SearchService();
const storageService = new StorageService();
const databaseService = new DatabaseService();
const indexingService = new IndexingService(databaseService, searchService, storageService);
await indexingService.indexByEmail(email, source.id, archivedEmail.id);
} catch (error) {
logger.error({
message: `Failed to process email ${email.id} for source ${source.id}`,
error,
emailId: email.id,
ingestionSourceId: source.id
});
}
}
} }

198
packages/backend/src/services/SearchService.ts
View File

@@ -3,116 +3,122 @@ import { config } from '../config';
import type { SearchQuery, SearchResult, EmailDocument, TopSender } from '@open-archiver/types'; import type { SearchQuery, SearchResult, EmailDocument, TopSender } from '@open-archiver/types';
export class SearchService { export class SearchService {
private client: MeiliSearch; private client: MeiliSearch;
constructor() { constructor() {
this.client = new MeiliSearch({ this.client = new MeiliSearch({
host: config.search.host, host: config.search.host,
apiKey: config.search.apiKey, apiKey: config.search.apiKey,
}); });
} }
public async getIndex<T extends Record<string, any>>(name: string): Promise<Index<T>> { public async getIndex<T extends Record<string, any>>(name: string): Promise<Index<T>> {
return this.client.index<T>(name); return this.client.index<T>(name);
} }
public async addDocuments<T extends Record<string, any>>( public async addDocuments<T extends Record<string, any>>(
indexName: string, indexName: string,
documents: T[], documents: T[],
primaryKey?: string primaryKey?: string
) { ) {
const index = await this.getIndex<T>(indexName); const index = await this.getIndex<T>(indexName);
if (primaryKey) { if (primaryKey) {
index.update({ primaryKey }); index.update({ primaryKey });
} }
return index.addDocuments(documents); return index.addDocuments(documents);
} }
public async search<T extends Record<string, any>>(indexName: string, query: string, options?: any) { public async search<T extends Record<string, any>>(
const index = await this.getIndex<T>(indexName); indexName: string,
return index.search(query, options); query: string,
} options?: any
) {
const index = await this.getIndex<T>(indexName);
return index.search(query, options);
}
public async deleteDocuments(indexName: string, ids: string[]) { public async deleteDocuments(indexName: string, ids: string[]) {
const index = await this.getIndex(indexName); const index = await this.getIndex(indexName);
return index.deleteDocuments(ids); return index.deleteDocuments(ids);
} }
public async deleteDocumentsByFilter(indexName: string, filter: string | string[]) { public async deleteDocumentsByFilter(indexName: string, filter: string | string[]) {
const index = await this.getIndex(indexName); const index = await this.getIndex(indexName);
return index.deleteDocuments({ filter }); return index.deleteDocuments({ filter });
} }
public async searchEmails(dto: SearchQuery): Promise<SearchResult> { public async searchEmails(dto: SearchQuery): Promise<SearchResult> {
const { query, filters, page = 1, limit = 10, matchingStrategy = 'last' } = dto; const { query, filters, page = 1, limit = 10, matchingStrategy = 'last' } = dto;
const index = await this.getIndex<EmailDocument>('emails'); const index = await this.getIndex<EmailDocument>('emails');
const searchParams: SearchParams = { const searchParams: SearchParams = {
limit, limit,
offset: (page - 1) * limit, offset: (page - 1) * limit,
attributesToHighlight: ['*'], attributesToHighlight: ['*'],
showMatchesPosition: true, showMatchesPosition: true,
sort: ['timestamp:desc'], sort: ['timestamp:desc'],
matchingStrategy matchingStrategy,
}; };
if (filters) { if (filters) {
const filterStrings = Object.entries(filters).map(([key, value]) => { const filterStrings = Object.entries(filters).map(([key, value]) => {
if (typeof value === 'string') { if (typeof value === 'string') {
return `${key} = '${value}'`; return `${key} = '${value}'`;
} }
return `${key} = ${value}`; return `${key} = ${value}`;
}); });
searchParams.filter = filterStrings.join(' AND '); searchParams.filter = filterStrings.join(' AND ');
} }
const searchResults = await index.search(query, searchParams); const searchResults = await index.search(query, searchParams);
return { return {
hits: searchResults.hits, hits: searchResults.hits,
total: searchResults.estimatedTotalHits ?? searchResults.hits.length, total: searchResults.estimatedTotalHits ?? searchResults.hits.length,
page, page,
limit, limit,
totalPages: Math.ceil((searchResults.estimatedTotalHits ?? searchResults.hits.length) / limit), totalPages: Math.ceil(
processingTimeMs: searchResults.processingTimeMs (searchResults.estimatedTotalHits ?? searchResults.hits.length) / limit
}; ),
} processingTimeMs: searchResults.processingTimeMs,
};
}
public async getTopSenders(limit = 10): Promise<TopSender[]> { public async getTopSenders(limit = 10): Promise<TopSender[]> {
const index = await this.getIndex<EmailDocument>('emails'); const index = await this.getIndex<EmailDocument>('emails');
const searchResults = await index.search('', { const searchResults = await index.search('', {
facets: ['from'], facets: ['from'],
limit: 0 limit: 0,
}); });
if (!searchResults.facetDistribution?.from) { if (!searchResults.facetDistribution?.from) {
return []; return [];
} }
// Sort and take top N // Sort and take top N
const sortedSenders = Object.entries(searchResults.facetDistribution.from) const sortedSenders = Object.entries(searchResults.facetDistribution.from)
.sort(([, countA], [, countB]) => countB - countA) .sort(([, countA], [, countB]) => countB - countA)
.slice(0, limit) .slice(0, limit)
.map(([sender, count]) => ({ sender, count })); .map(([sender, count]) => ({ sender, count }));
return sortedSenders; return sortedSenders;
} }
public async configureEmailIndex() { public async configureEmailIndex() {
const index = await this.getIndex('emails'); const index = await this.getIndex('emails');
await index.updateSettings({ await index.updateSettings({
searchableAttributes: [ searchableAttributes: [
'subject', 'subject',
'body', 'body',
'from', 'from',
'to', 'to',
'cc', 'cc',
'bcc', 'bcc',
'attachments.filename', 'attachments.filename',
'attachments.content', 'attachments.content',
], ],
filterableAttributes: ['from', 'to', 'cc', 'bcc', 'timestamp', 'ingestionSourceId'], filterableAttributes: ['from', 'to', 'cc', 'bcc', 'timestamp', 'ingestionSourceId'],
sortableAttributes: ['timestamp'] sortableAttributes: ['timestamp'],
}); });
} }
} }

50
packages/backend/src/services/StorageService.ts
View File

@@ -4,34 +4,34 @@ import { S3StorageProvider } from './storage/S3StorageProvider';
import { config } from '../config/index'; import { config } from '../config/index';
export class StorageService implements IStorageProvider { export class StorageService implements IStorageProvider {
private provider: IStorageProvider; private provider: IStorageProvider;
constructor(storageConfig: StorageConfig = config.storage) { constructor(storageConfig: StorageConfig = config.storage) {
switch (storageConfig.type) { switch (storageConfig.type) {
case 'local': case 'local':
this.provider = new LocalFileSystemProvider(storageConfig); this.provider = new LocalFileSystemProvider(storageConfig);
break; break;
case 's3': case 's3':
this.provider = new S3StorageProvider(storageConfig); this.provider = new S3StorageProvider(storageConfig);
break; break;
default: default:
throw new Error('Invalid storage provider type'); throw new Error('Invalid storage provider type');
} }
} }
put(path: string, content: Buffer | NodeJS.ReadableStream): Promise<void> { put(path: string, content: Buffer | NodeJS.ReadableStream): Promise<void> {
return this.provider.put(path, content); return this.provider.put(path, content);
} }
get(path: string): Promise<NodeJS.ReadableStream> { get(path: string): Promise<NodeJS.ReadableStream> {
return this.provider.get(path); return this.provider.get(path);
} }
delete(path: string): Promise<void> { delete(path: string): Promise<void> {
return this.provider.delete(path); return this.provider.delete(path);
} }
exists(path: string): Promise<boolean> { exists(path: string): Promise<boolean> {
return this.provider.exists(path); return this.provider.exists(path);
} }
} }

Some files were not shown because too many files have changed in this diff Show More