From 42e32f75bc99b4a71aaca4ef658566fcc45a7d64 Mon Sep 17 00:00:00 2001 From: wayneshn Date: Sun, 27 Jul 2025 22:33:25 +0000 Subject: [PATCH] deploy: 9d3e6fc22e22a2b41eea2ac8365deaab7f689ac1 --- 404.html | 4 ++-- CNAME | 1 + SUMMARY.html | 8 ++++---- api/index.html | 8 ++++---- api/ingestion.html | 8 ++++---- index.html | 8 ++++---- services/index.html | 8 ++++---- services/storage-service.html | 8 ++++---- user-guides/email-providers/google-workspace.html | 8 ++++---- user-guides/email-providers/imap.html | 8 ++++---- user-guides/email-providers/index.html | 8 ++++---- user-guides/email-providers/microsoft-365.html | 8 ++++---- user-guides/installation.html | 8 ++++---- 13 files changed, 47 insertions(+), 46 deletions(-) create mode 100644 CNAME diff --git a/404.html b/404.html index e1e455b..cc7fc7d 100644 --- a/404.html +++ b/404.html @@ -3,7 +3,7 @@ - 404 | OpenArchiver Documentation + 404 | Open Archiver Documentation @@ -16,7 +16,7 @@
- + \ No newline at end of file diff --git a/CNAME b/CNAME new file mode 100644 index 0000000..db07058 --- /dev/null +++ b/CNAME @@ -0,0 +1 @@ +docs.openarchiver.com diff --git a/SUMMARY.html b/SUMMARY.html index 335012c..7de7a1d 100644 --- a/SUMMARY.html +++ b/SUMMARY.html @@ -3,8 +3,8 @@ - Table of contents | OpenArchiver Documentation - + Table of contents | Open Archiver Documentation + @@ -18,8 +18,8 @@ -
- +
+ \ No newline at end of file diff --git a/api/index.html b/api/index.html index f850d0b..ba350c8 100644 --- a/api/index.html +++ b/api/index.html @@ -3,8 +3,8 @@ - API documentation | OpenArchiver Documentation - + API documentation | Open Archiver Documentation + @@ -18,8 +18,8 @@ -
- +
+ \ No newline at end of file diff --git a/api/ingestion.html b/api/ingestion.html index 73f7fa7..2fc5c33 100644 --- a/api/ingestion.html +++ b/api/ingestion.html @@ -3,8 +3,8 @@ - Ingestion Sources API Documentation | OpenArchiver Documentation - + Ingestion Sources API Documentation | Open Archiver Documentation + @@ -18,7 +18,7 @@ -
Skip to content

Ingestion Sources API Documentation

A guide to using the Ingestion Sources API.

Base Path: /v1/ingestion-sources


Authentication

All endpoints in this API are protected and require authentication. Requests must include an Authorization header containing a valid Bearer token. This can be a JWT obtained from the login endpoint or a SUPER_API_KEY for administrative tasks.

Header Example:Authorization: Bearer <YOUR_JWT_OR_SUPER_API_KEY>


Core Concepts

Ingestion Providers

The provider field determines the type of email source. Each provider requires a different configuration object, for example:

  • google_workspace: For connecting to Google Workspace accounts via OAuth 2.0.
  • microsoft_365: For connecting to Microsoft 365 accounts via OAuth 2.0.
  • generic_imap: For connecting to any email server that supports IMAP.

Ingestion Status

The status field tracks the state of the ingestion source.

  • pending_auth: The source has been created but requires user authorization (OAuth flow).
  • active: The source is authenticated and ready to sync.
  • syncing: An import job is currently in progress.
  • importing: initial syncing in progress
  • paused: The source is temporarily disabled.
  • error: An error occurred during the last sync.

1. Create Ingestion Source

  • Method: POST
  • Path: /
  • Description: Registers a new source for email ingestion. The providerConfig will vary based on the selected provider.

Request Body (CreateIngestionSourceDto)

  • name (string, required): A user-friendly name for the source (e.g., "Marketing Department G-Suite").
  • provider (string, required): One of google_workspace, microsoft_365, or generic_imap.
  • providerConfig (object, required): Configuration specific to the provider.
providerConfig for google_workspace / microsoft_365
json
{
+    
Skip to content

Ingestion Sources API Documentation

A guide to using the Ingestion Sources API.

Base Path: /v1/ingestion-sources


Authentication

All endpoints in this API are protected and require authentication. Requests must include an Authorization header containing a valid Bearer token. This can be a JWT obtained from the login endpoint or a SUPER_API_KEY for administrative tasks.

Header Example:Authorization: Bearer <YOUR_JWT_OR_SUPER_API_KEY>


Core Concepts

Ingestion Providers

The provider field determines the type of email source. Each provider requires a different configuration object, for example:

  • google_workspace: For connecting to Google Workspace accounts via OAuth 2.0.
  • microsoft_365: For connecting to Microsoft 365 accounts via OAuth 2.0.
  • generic_imap: For connecting to any email server that supports IMAP.

Ingestion Status

The status field tracks the state of the ingestion source.

  • pending_auth: The source has been created but requires user authorization (OAuth flow).
  • active: The source is authenticated and ready to sync.
  • syncing: An import job is currently in progress.
  • importing: initial syncing in progress
  • paused: The source is temporarily disabled.
  • error: An error occurred during the last sync.

1. Create Ingestion Source

  • Method: POST
  • Path: /
  • Description: Registers a new source for email ingestion. The providerConfig will vary based on the selected provider.

Request Body (CreateIngestionSourceDto)

  • name (string, required): A user-friendly name for the source (e.g., "Marketing Department G-Suite").
  • provider (string, required): One of google_workspace, microsoft_365, or generic_imap.
  • providerConfig (object, required): Configuration specific to the provider.
providerConfig for google_workspace / microsoft_365
json
{
     "name": "Corporate Google Workspace",
     "provider": "google_workspace",
     "providerConfig": {
@@ -50,7 +50,7 @@
 }

Responses

  • Success (200 OK): Returns the complete, updated IngestionSource object.

  • Error (404 Not Found): Returned if no source with the given ID exists.

  • Error (500 Internal Server Error): Indicates a server-side problem.


5. Delete Ingestion Source

  • Method: DELETE
  • Path: /:id
  • Description: Permanently removes an ingestion source. This action cannot be undone.

URL Parameters

  • id (string, required): The UUID of the ingestion source to delete.

Responses

  • Success (204 No Content): Indicates successful deletion with no body content.

  • Error (404 Not Found): Returned if no source with the given ID exists.

  • Error (500 Internal Server Error): Indicates a server-side problem.


6. Trigger Initial Import

  • Method: POST
  • Path: /:id/sync
  • Description: Initiates the email import process for a given source. This is an asynchronous operation that enqueues a background job and immediately returns a response. The status of the source will be updated to importing.

URL Parameters

  • id (string, required): The UUID of the ingestion source to sync.

Responses

  • Success (202 Accepted): Confirms that the sync request has been accepted for processing.

    json
    {
         "message": "Initial import triggered successfully."
     }
  • Error (404 Not Found): Returned if no source with the given ID exists.

  • Error (500 Internal Server Error): Indicates a server-side problem.

- + \ No newline at end of file diff --git a/index.html b/index.html index 2317e2e..1c38bfd 100644 --- a/index.html +++ b/index.html @@ -3,8 +3,8 @@ - Get Started | OpenArchiver Documentation - + Get Started | Open Archiver Documentation + @@ -18,8 +18,8 @@ -
Skip to content

Get Started

Welcome to Open Archiver! This guide will help you get started with setting up and using the platform.

What is Open Archiver?

A secure, sovereign, and affordable open-source platform for email archiving and eDiscovery.

Open Archiver provides a robust, self-hosted solution for archiving, storing, indexing, and searching emails from major platforms, including Google Workspace (Gmail), Microsoft 365, as well as generic IMAP-enabled email inboxes. Use Open Archiver to keep a permanent, tamper-proof record of your communication history, free from vendor lock-in.

Key Features

  • Universal Ingestion: Connect to Google Workspace, Microsoft 365, and standard IMAP servers to perform initial bulk imports and maintain continuous, real-time synchronization.
  • Secure & Efficient Storage: Emails are stored in the standard .eml format. The system uses deduplication and compression to minimize storage costs. All data is encrypted at rest.
  • Pluggable Storage Backends: Support both local filesystem storage and S3-compatible object storage (like AWS S3 or MinIO).
  • Powerful Search & eDiscovery: A high-performance search engine indexes the full text of emails and attachments (PDF, DOCX, etc.).
  • Compliance & Retention: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD).
  • Comprehensive Auditing: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when (TBD).

Installation

To get your own instance of Open Archiver running, follow our detailed installation guide:

Data Source Configuration

After deploying the application, you will need to configure one or more ingestion sources to begin archiving emails. Follow our detailed guides to connect to your email provider:

Contributing

We welcome contributions from the community!

  • Reporting Bugs: If you find a bug, please open an issue on our GitHub repository.
  • Suggesting Enhancements: Have an idea for a new feature? We'd love to hear it. Open an issue to start the discussion.
  • Code Contributions: If you'd like to contribute code, please fork the repository and submit a pull request.

Please read our CONTRIBUTING.md file for more details on our code of conduct and the process for submitting pull requests.

- +
Skip to content

Get Started

Welcome to Open Archiver! This guide will help you get started with setting up and using the platform.

What is Open Archiver?

A secure, sovereign, and affordable open-source platform for email archiving and eDiscovery.

Open Archiver provides a robust, self-hosted solution for archiving, storing, indexing, and searching emails from major platforms, including Google Workspace (Gmail), Microsoft 365, as well as generic IMAP-enabled email inboxes. Use Open Archiver to keep a permanent, tamper-proof record of your communication history, free from vendor lock-in.

Key Features

  • Universal Ingestion: Connect to Google Workspace, Microsoft 365, and standard IMAP servers to perform initial bulk imports and maintain continuous, real-time synchronization.
  • Secure & Efficient Storage: Emails are stored in the standard .eml format. The system uses deduplication and compression to minimize storage costs. All data is encrypted at rest.
  • Pluggable Storage Backends: Support both local filesystem storage and S3-compatible object storage (like AWS S3 or MinIO).
  • Powerful Search & eDiscovery: A high-performance search engine indexes the full text of emails and attachments (PDF, DOCX, etc.).
  • Compliance & Retention: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD).
  • Comprehensive Auditing: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when (TBD).

Installation

To get your own instance of Open Archiver running, follow our detailed installation guide:

Data Source Configuration

After deploying the application, you will need to configure one or more ingestion sources to begin archiving emails. Follow our detailed guides to connect to your email provider:

Contributing

We welcome contributions from the community!

  • Reporting Bugs: If you find a bug, please open an issue on our GitHub repository.
  • Suggesting Enhancements: Have an idea for a new feature? We'd love to hear it. Open an issue to start the discussion.
  • Code Contributions: If you'd like to contribute code, please fork the repository and submit a pull request.

Please read our CONTRIBUTING.md file for more details on our code of conduct and the process for submitting pull requests.

+ \ No newline at end of file diff --git a/services/index.html b/services/index.html index 57d97db..9a4e364 100644 --- a/services/index.html +++ b/services/index.html @@ -3,8 +3,8 @@ - services | OpenArchiver Documentation - + services | Open Archiver Documentation + @@ -18,8 +18,8 @@ - - + + \ No newline at end of file diff --git a/services/storage-service.html b/services/storage-service.html index 96a02ab..c374e56 100644 --- a/services/storage-service.html +++ b/services/storage-service.html @@ -3,8 +3,8 @@ - Pluggable Storage Service (StorageService) | OpenArchiver Documentation - + Pluggable Storage Service (StorageService) | Open Archiver Documentation + @@ -18,7 +18,7 @@ -
Skip to content

Pluggable Storage Service (StorageService)

Overview

The StorageService provides a unified, abstract interface for handling file storage across different backends. Its primary purpose is to decouple the application's core logic from the underlying storage technology. This design allows administrators to switch between storage providers (e.g., from the local filesystem to an S3-compatible object store) with only a configuration change, requiring no modifications to the application code.

The service is built around a standardized IStorageProvider interface, which guarantees that all storage providers have a consistent API for common operations like storing, retrieving, and deleting files.

Configuration

The StorageService is configured via environment variables in the .env file. You must specify the storage backend you wish to use and provide the necessary credentials and settings for it.

1. Choosing the Backend

The STORAGE_TYPE variable determines which provider the service will use.

  • STORAGE_TYPE=local: Uses the local server's filesystem.
  • STORAGE_TYPE=s3: Uses an S3-compatible object storage service (e.g., AWS S3, MinIO, Google Cloud Storage).

2. Local Filesystem Configuration

When STORAGE_TYPE is set to local, you must also provide the root path where files will be stored.

env
# .env
+    
Skip to content

Pluggable Storage Service (StorageService)

Overview

The StorageService provides a unified, abstract interface for handling file storage across different backends. Its primary purpose is to decouple the application's core logic from the underlying storage technology. This design allows administrators to switch between storage providers (e.g., from the local filesystem to an S3-compatible object store) with only a configuration change, requiring no modifications to the application code.

The service is built around a standardized IStorageProvider interface, which guarantees that all storage providers have a consistent API for common operations like storing, retrieving, and deleting files.

Configuration

The StorageService is configured via environment variables in the .env file. You must specify the storage backend you wish to use and provide the necessary credentials and settings for it.

1. Choosing the Backend

The STORAGE_TYPE variable determines which provider the service will use.

  • STORAGE_TYPE=local: Uses the local server's filesystem.
  • STORAGE_TYPE=s3: Uses an S3-compatible object storage service (e.g., AWS S3, MinIO, Google Cloud Storage).

2. Local Filesystem Configuration

When STORAGE_TYPE is set to local, you must also provide the root path where files will be stored.

env
# .env
 STORAGE_TYPE=local
 STORAGE_LOCAL_ROOT_PATH=/var/data/open-archiver
  • STORAGE_LOCAL_ROOT_PATH: The absolute path on the server where the archive will be created. The service will create subdirectories within this path as needed.

3. S3-Compatible Storage Configuration

When STORAGE_TYPE is set to s3, you must provide the credentials and endpoint for your object storage provider.

env
# .env
 STORAGE_TYPE=s3
@@ -56,7 +56,7 @@
         }
     }
 }

API Reference

The StorageService implements the IStorageProvider interface. All methods are asynchronous and return a Promise.


put(path, content)

Stores a file at the specified path. If a file already exists at that path, it will be overwritten.

  • path: string: A unique identifier for the file, including its directory structure (e.g., "user-123/emails/message-abc.eml").
  • content: Buffer | NodeJS.ReadableStream: The content of the file. It can be a Buffer for small files or a ReadableStream for large files to ensure memory efficiency.
  • Returns: Promise<void> - A promise that resolves when the file has been successfully stored.

get(path)

Retrieves a file from the specified path as a readable stream.

  • path: string: The unique identifier of the file to retrieve.
  • Returns: Promise<NodeJS.ReadableStream> - A promise that resolves with a readable stream of the file's content.
  • Throws: An Error if the file is not found at the specified path.

delete(path)

Deletes a file from the storage backend.

  • path: string: The unique identifier of the file to delete.
  • Returns: Promise<void> - A promise that resolves when the file is deleted. If the file does not exist, the promise will still resolve successfully without throwing an error.

exists(path)

Checks for the existence of a file.

  • path: string: The unique identifier of the file to check.
  • Returns: Promise<boolean> - A promise that resolves with true if the file exists, and false otherwise.
- + \ No newline at end of file diff --git a/user-guides/email-providers/google-workspace.html b/user-guides/email-providers/google-workspace.html index 55c527d..1223ab7 100644 --- a/user-guides/email-providers/google-workspace.html +++ b/user-guides/email-providers/google-workspace.html @@ -3,8 +3,8 @@ - Connecting to Google Workspace | OpenArchiver Documentation - + Connecting to Google Workspace | Open Archiver Documentation + @@ -18,8 +18,8 @@ -
Skip to content

Connecting to Google Workspace

This guide provides instructions for Google Workspace administrators to set up a connection that allows the archiving of all user mailboxes within their organization.

The connection uses a Google Cloud Service Account with Domain-Wide Delegation. This is a secure method that grants the archiving service permission to access user data on behalf of the administrator, without requiring individual user passwords or consent.

Prerequisites

  • You must have Super Administrator privileges in your Google Workspace account.
  • You must have access to the Google Cloud Console associated with your organization.

Setup Overview

The setup process involves three main parts:

  1. Configuring the necessary permissions in the Google Cloud Console.
  2. Authorizing the service account in the Google Workspace Admin Console.
  3. Entering the generated credentials into the OpenArchiver application.

Part 1: Google Cloud Console Setup

In this part, you will create a service account and enable the APIs it needs to function.

  1. Create a Google Cloud Project:

    • Go to the Google Cloud Console.
    • If you don't already have one, create a new project for the archiving service (e.g., "Email Archiver").
  2. Enable Required APIs:

    • In your selected project, navigate to the "APIs & Services" > "Library" section.
    • Search for and enable the following two APIs:
      • Gmail API
      • Admin SDK API
  3. Create a Service Account:

    • Navigate to "IAM & Admin" > "Service Accounts".
    • Click "Create Service Account".
    • Give the service account a name (e.g., email-archiver-service) and a description.
    • Click "Create and Continue". You do not need to grant this service account any roles on the project. Click "Done".
  4. Generate a JSON Key:

    • Find the service account you just created in the list.
    • Click the three-dot menu under "Actions" and select "Manage keys".
    • Click "Add Key" > "Create new key".
    • Select JSON as the key type and click "Create".
    • A JSON file will be downloaded to your computer. Keep this file secure, as it contains private credentials. You will need the contents of this file in Part 3.

Troubleshooting

Error: "iam.disableServiceAccountKeyCreation"

If you receive an error message stating The organization policy constraint 'iam.disableServiceAccountKeyCreation' is enforced when trying to create a JSON key, it means your Google Cloud organization has a policy preventing the creation of new service account keys.

To resolve this, you must have Organization Administrator permissions.

  1. Navigate to your Organization: In the Google Cloud Console, use the project selector at the top of the page to select your organization node (it usually has a building icon).
  2. Go to IAM: From the navigation menu, select "IAM & Admin" > "IAM".
  3. Edit Your Permissions: Find your user account in the list and click the pencil icon to edit roles. Add the following two roles:
    • Organization Policy Administrator
    • Organization AdministratorNote: These roles are only available at the organization level, not the project level.
  4. Modify the Policy:
    • Navigate to "IAM & Admin" > "Organization Policies".
    • In the filter box, search for the policy "iam.disableServiceAccountKeyCreation".
    • Click on the policy to edit it.
    • You can either disable the policy entirely (if your security rules permit) or add a rule to exclude the specific project you are using for the archiver from this policy.
  5. Retry Key Creation: Once the policy is updated, return to your project and you should be able to generate the JSON key as described in Part 1.

Part 2: Grant Domain-Wide Delegation

Now, you will authorize the service account you created to access data from your Google Workspace.

  1. Get the Service Account's Client ID:

    • Go back to the list of service accounts in the Google Cloud Console.
    • Click on the service account you created.
    • Under the "Details" tab, find and copy the Unique ID (this is the Client ID).
  2. Authorize the Client in Google Workspace:

    • Go to your Google Workspace Admin Console at admin.google.com.
    • Navigate to Security > Access and data control > API controls.
    • Under the "Domain-wide Delegation" section, click "Manage Domain-wide Delegation".
    • Click "Add new".
  3. Enter Client Details and Scopes:

    • In the Client ID field, paste the Unique ID you copied from the service account.
    • In the OAuth scopes field, paste the following two scopes exactly as they appear, separated by a comma:
      https://www.googleapis.com/auth/admin.directory.user.readonly,https://www.googleapis.com/auth/gmail.readonly
    • Click "Authorize".

The service account is now permitted to list users and read their email data across your domain.


Part 3: Connecting in OpenArchiver

Finally, you will provide the generated credentials to the application.

  1. Navigate to Ingestion Sources: From the main dashboard, go to the Ingestion Sources page.

  2. Create a New Source: Click the "Create New" button.

  3. Fill in the Configuration Details:

    • Name: Give the source a name (e.g., "Google Workspace Archive").
    • Provider: Select "Google Workspace" from the dropdown.
    • Service Account Key (JSON): Open the JSON file you downloaded in Part 1. Copy the entire content of the file and paste it into this text area.
    • Impersonated Admin Email: Enter the email address of a Super Administrator in your Google Workspace (e.g., admin@your-domain.com). The service will use this user's authority to discover all other users.
  4. Save Changes: Click "Save changes".

What Happens Next?

Once the connection is saved and verified, the system will begin the archiving process:

  1. User Discovery: The service will first connect to the Admin SDK to get a list of all active users in your Google Workspace.
  2. Initial Import: The system will then start a background job to import the mailboxes of all discovered users. The status will show as "Importing". This can take a significant amount of time depending on the number of users and the size of their mailboxes.
  3. Continuous Sync: After the initial import is complete, the status will change to "Active". The system will then periodically check each user's mailbox for new emails and archive them automatically.
- +
Skip to content

Connecting to Google Workspace

This guide provides instructions for Google Workspace administrators to set up a connection that allows the archiving of all user mailboxes within their organization.

The connection uses a Google Cloud Service Account with Domain-Wide Delegation. This is a secure method that grants the archiving service permission to access user data on behalf of the administrator, without requiring individual user passwords or consent.

Prerequisites

  • You must have Super Administrator privileges in your Google Workspace account.
  • You must have access to the Google Cloud Console associated with your organization.

Setup Overview

The setup process involves three main parts:

  1. Configuring the necessary permissions in the Google Cloud Console.
  2. Authorizing the service account in the Google Workspace Admin Console.
  3. Entering the generated credentials into the OpenArchiver application.

Part 1: Google Cloud Console Setup

In this part, you will create a service account and enable the APIs it needs to function.

  1. Create a Google Cloud Project:

    • Go to the Google Cloud Console.
    • If you don't already have one, create a new project for the archiving service (e.g., "Email Archiver").
  2. Enable Required APIs:

    • In your selected project, navigate to the "APIs & Services" > "Library" section.
    • Search for and enable the following two APIs:
      • Gmail API
      • Admin SDK API
  3. Create a Service Account:

    • Navigate to "IAM & Admin" > "Service Accounts".
    • Click "Create Service Account".
    • Give the service account a name (e.g., email-archiver-service) and a description.
    • Click "Create and Continue". You do not need to grant this service account any roles on the project. Click "Done".
  4. Generate a JSON Key:

    • Find the service account you just created in the list.
    • Click the three-dot menu under "Actions" and select "Manage keys".
    • Click "Add Key" > "Create new key".
    • Select JSON as the key type and click "Create".
    • A JSON file will be downloaded to your computer. Keep this file secure, as it contains private credentials. You will need the contents of this file in Part 3.

Troubleshooting

Error: "iam.disableServiceAccountKeyCreation"

If you receive an error message stating The organization policy constraint 'iam.disableServiceAccountKeyCreation' is enforced when trying to create a JSON key, it means your Google Cloud organization has a policy preventing the creation of new service account keys.

To resolve this, you must have Organization Administrator permissions.

  1. Navigate to your Organization: In the Google Cloud Console, use the project selector at the top of the page to select your organization node (it usually has a building icon).
  2. Go to IAM: From the navigation menu, select "IAM & Admin" > "IAM".
  3. Edit Your Permissions: Find your user account in the list and click the pencil icon to edit roles. Add the following two roles:
    • Organization Policy Administrator
    • Organization AdministratorNote: These roles are only available at the organization level, not the project level.
  4. Modify the Policy:
    • Navigate to "IAM & Admin" > "Organization Policies".
    • In the filter box, search for the policy "iam.disableServiceAccountKeyCreation".
    • Click on the policy to edit it.
    • You can either disable the policy entirely (if your security rules permit) or add a rule to exclude the specific project you are using for the archiver from this policy.
  5. Retry Key Creation: Once the policy is updated, return to your project and you should be able to generate the JSON key as described in Part 1.

Part 2: Grant Domain-Wide Delegation

Now, you will authorize the service account you created to access data from your Google Workspace.

  1. Get the Service Account's Client ID:

    • Go back to the list of service accounts in the Google Cloud Console.
    • Click on the service account you created.
    • Under the "Details" tab, find and copy the Unique ID (this is the Client ID).
  2. Authorize the Client in Google Workspace:

    • Go to your Google Workspace Admin Console at admin.google.com.
    • Navigate to Security > Access and data control > API controls.
    • Under the "Domain-wide Delegation" section, click "Manage Domain-wide Delegation".
    • Click "Add new".
  3. Enter Client Details and Scopes:

    • In the Client ID field, paste the Unique ID you copied from the service account.
    • In the OAuth scopes field, paste the following two scopes exactly as they appear, separated by a comma:
      https://www.googleapis.com/auth/admin.directory.user.readonly,https://www.googleapis.com/auth/gmail.readonly
    • Click "Authorize".

The service account is now permitted to list users and read their email data across your domain.


Part 3: Connecting in OpenArchiver

Finally, you will provide the generated credentials to the application.

  1. Navigate to Ingestion Sources: From the main dashboard, go to the Ingestion Sources page.

  2. Create a New Source: Click the "Create New" button.

  3. Fill in the Configuration Details:

    • Name: Give the source a name (e.g., "Google Workspace Archive").
    • Provider: Select "Google Workspace" from the dropdown.
    • Service Account Key (JSON): Open the JSON file you downloaded in Part 1. Copy the entire content of the file and paste it into this text area.
    • Impersonated Admin Email: Enter the email address of a Super Administrator in your Google Workspace (e.g., admin@your-domain.com). The service will use this user's authority to discover all other users.
  4. Save Changes: Click "Save changes".

What Happens Next?

Once the connection is saved and verified, the system will begin the archiving process:

  1. User Discovery: The service will first connect to the Admin SDK to get a list of all active users in your Google Workspace.
  2. Initial Import: The system will then start a background job to import the mailboxes of all discovered users. The status will show as "Importing". This can take a significant amount of time depending on the number of users and the size of their mailboxes.
  3. Continuous Sync: After the initial import is complete, the status will change to "Active". The system will then periodically check each user's mailbox for new emails and archive them automatically.
+ \ No newline at end of file diff --git a/user-guides/email-providers/imap.html b/user-guides/email-providers/imap.html index f1c3e8a..a57a841 100644 --- a/user-guides/email-providers/imap.html +++ b/user-guides/email-providers/imap.html @@ -3,8 +3,8 @@ - Connecting to a Generic IMAP Server | OpenArchiver Documentation - + Connecting to a Generic IMAP Server | Open Archiver Documentation + @@ -18,8 +18,8 @@ -
Skip to content

Connecting to a Generic IMAP Server

This guide will walk you through connecting a standard IMAP email account as an ingestion source. This allows you to archive emails from any provider that supports the IMAP protocol, which is common for many self-hosted or traditional email services.

Step-by-Step Guide

  1. Navigate to Ingestion Sources: From the main dashboard, go to the Ingestions page.

  2. Create a New Source: Click the "Create New" button to open the ingestion source configuration dialog.

  3. Fill in the Configuration Details: You will see a form with several fields. Here is how to fill them out for an IMAP connection:

    • Name: Give your ingestion source a descriptive name that you will easily recognize, such as "Work Email (IMAP)" or "Personal Gmail".

    • Provider: From the dropdown menu, select "Generic IMAP". This will reveal the specific fields required for an IMAP connection.

    • Host: Enter the server address for your email provider's IMAP service. This often looks like imap.your-provider.com or mail.your-domain.com.

    • Port: Enter the port number for the IMAP server. For a secure connection (which is strongly recommended), this is typically 993.

    • Username: Enter the full email address or username you use to log in to your email account.

    • Password: Enter the password for your email account.

  4. Save Changes: Once you have filled in all the details, click the "Save changes" button.

Security Recommendation: Use an App Password

For enhanced security, we strongly recommend using an "app password" (sometimes called an "app-specific password") instead of your main account password.

Many email providers (like Gmail, Outlook, and Fastmail) allow you to generate a unique password that grants access only to a specific application (in this case, the archiving service). If you ever need to revoke access, you can simply delete the app password without affecting your main account login.

Please consult your email provider's documentation to see if they support app passwords and how to create one.

How to Obtain an App Password for Gmail

  1. Enable 2-Step Verification: You must have 2-Step Verification turned on for your Google Account.
  2. Go to App Passwords: Visit myaccount.google.com/apppasswords. You may be asked to sign in again.
  3. Create the Password:
    • At the bottom, click "Select app" and choose "Other (Custom name)".
    • Give it a name you'll recognize, like "OpenArchiver".
    • Click "Generate".
  4. Use the Password: A 16-digit password will be displayed. Copy this password (without the spaces) and paste it into the Password field in the OpenArchiver ingestion source form.

How to Obtain an App Password for Outlook/Microsoft Accounts

  1. Enable Two-Step Verification: You must have two-step verification enabled for your Microsoft account.
  2. Go to Security Options: Sign in to your Microsoft account and navigate to the Advanced security options.
  3. Create a New App Password:
    • Scroll down to the "App passwords" section.
    • Click "Create a new app password".
  4. Use the Password: A new password will be generated. Use this password in the Password field in the OpenArchiver ingestion source form.

What Happens Next?

After you save the connection, the system will attempt to connect to the IMAP server. The status of the ingestion source will update to reflect its current state:

  • Importing: The system is performing the initial, one-time import of all emails from your INBOX. This may take a while depending on the size of your mailbox.
  • Active: The initial import is complete, and the system will now periodically check for and archive new emails.
  • Paused: The connection is valid, but the system will not check for new emails until you resume it.
  • Error: The system was unable to connect using the provided credentials. Please double-check your Host, Port, Username, and Password and try again.

You can view, edit, pause, or manually sync any of your ingestion sources from the main table on the Ingestions page.

- +
Skip to content

Connecting to a Generic IMAP Server

This guide will walk you through connecting a standard IMAP email account as an ingestion source. This allows you to archive emails from any provider that supports the IMAP protocol, which is common for many self-hosted or traditional email services.

Step-by-Step Guide

  1. Navigate to Ingestion Sources: From the main dashboard, go to the Ingestions page.

  2. Create a New Source: Click the "Create New" button to open the ingestion source configuration dialog.

  3. Fill in the Configuration Details: You will see a form with several fields. Here is how to fill them out for an IMAP connection:

    • Name: Give your ingestion source a descriptive name that you will easily recognize, such as "Work Email (IMAP)" or "Personal Gmail".

    • Provider: From the dropdown menu, select "Generic IMAP". This will reveal the specific fields required for an IMAP connection.

    • Host: Enter the server address for your email provider's IMAP service. This often looks like imap.your-provider.com or mail.your-domain.com.

    • Port: Enter the port number for the IMAP server. For a secure connection (which is strongly recommended), this is typically 993.

    • Username: Enter the full email address or username you use to log in to your email account.

    • Password: Enter the password for your email account.

  4. Save Changes: Once you have filled in all the details, click the "Save changes" button.

Security Recommendation: Use an App Password

For enhanced security, we strongly recommend using an "app password" (sometimes called an "app-specific password") instead of your main account password.

Many email providers (like Gmail, Outlook, and Fastmail) allow you to generate a unique password that grants access only to a specific application (in this case, the archiving service). If you ever need to revoke access, you can simply delete the app password without affecting your main account login.

Please consult your email provider's documentation to see if they support app passwords and how to create one.

How to Obtain an App Password for Gmail

  1. Enable 2-Step Verification: You must have 2-Step Verification turned on for your Google Account.
  2. Go to App Passwords: Visit myaccount.google.com/apppasswords. You may be asked to sign in again.
  3. Create the Password:
    • At the bottom, click "Select app" and choose "Other (Custom name)".
    • Give it a name you'll recognize, like "OpenArchiver".
    • Click "Generate".
  4. Use the Password: A 16-digit password will be displayed. Copy this password (without the spaces) and paste it into the Password field in the OpenArchiver ingestion source form.

How to Obtain an App Password for Outlook/Microsoft Accounts

  1. Enable Two-Step Verification: You must have two-step verification enabled for your Microsoft account.
  2. Go to Security Options: Sign in to your Microsoft account and navigate to the Advanced security options.
  3. Create a New App Password:
    • Scroll down to the "App passwords" section.
    • Click "Create a new app password".
  4. Use the Password: A new password will be generated. Use this password in the Password field in the OpenArchiver ingestion source form.

What Happens Next?

After you save the connection, the system will attempt to connect to the IMAP server. The status of the ingestion source will update to reflect its current state:

  • Importing: The system is performing the initial, one-time import of all emails from your INBOX. This may take a while depending on the size of your mailbox.
  • Active: The initial import is complete, and the system will now periodically check for and archive new emails.
  • Paused: The connection is valid, but the system will not check for new emails until you resume it.
  • Error: The system was unable to connect using the provided credentials. Please double-check your Host, Port, Username, and Password and try again.

You can view, edit, pause, or manually sync any of your ingestion sources from the main table on the Ingestions page.

+ \ No newline at end of file diff --git a/user-guides/email-providers/index.html b/user-guides/email-providers/index.html index dbb2267..c06521e 100644 --- a/user-guides/email-providers/index.html +++ b/user-guides/email-providers/index.html @@ -3,8 +3,8 @@ - Connecting Email Providers | OpenArchiver Documentation - + Connecting Email Providers | Open Archiver Documentation + @@ -18,8 +18,8 @@ -
Skip to content

Connecting Email Providers

Open Archiver can connect to a variety of email sources to ingest and archive your emails. This section provides guides for connecting to popular email providers.

Choose your provider from the list below to get started:

- +
Skip to content

Connecting Email Providers

Open Archiver can connect to a variety of email sources to ingest and archive your emails. This section provides guides for connecting to popular email providers.

Choose your provider from the list below to get started:

+ \ No newline at end of file diff --git a/user-guides/email-providers/microsoft-365.html b/user-guides/email-providers/microsoft-365.html index deb294a..b7f736b 100644 --- a/user-guides/email-providers/microsoft-365.html +++ b/user-guides/email-providers/microsoft-365.html @@ -3,8 +3,8 @@ - Connecting to Microsoft 365 | OpenArchiver Documentation - + Connecting to Microsoft 365 | Open Archiver Documentation + @@ -18,8 +18,8 @@ -
Skip to content

Connecting to Microsoft 365

This guide provides instructions for Microsoft 365 administrators to set up a connection that allows the archiving of all user mailboxes within their organization.

The connection uses the Microsoft Graph API and an App Registration in Microsoft Entra ID. This is a secure, standard method that grants the archiving service permission to read email data on your behalf without ever needing to handle user passwords.

Prerequisites

  • You must have one of the following administrator roles in your Microsoft 365 tenant: Global Administrator, Application Administrator, or Cloud Application Administrator.

Setup Overview

The setup process involves four main parts, all performed within the Microsoft Entra admin center and the OpenArchiver application:

  1. Registering a new application identity for the archiver in Entra ID.
  2. Granting the application the specific permissions it needs to read mail.
  3. Creating a secure password (a client secret) for the application.
  4. Entering the generated credentials into the OpenArchiver application.

Part 1: Register a New Application in Microsoft Entra ID

First, you will create an "App registration," which acts as an identity for the archiving service within your Microsoft 365 ecosystem.

  1. Sign in to the Microsoft Entra admin center.
  2. In the left-hand navigation pane, go to Identity > Applications > App registrations.
  3. Click the + New registration button at the top of the page.
  4. On the "Register an application" screen:
    • Name: Give the application a descriptive name you will recognize, such as OpenArchiver Service.
    • Supported account types: Select "Accounts in this organizational directory only (Default Directory only - Single tenant)". This is the most secure option.
    • Redirect URI (optional): You can leave this blank.
  5. Click the Register button. You will be taken to the application's main "Overview" page.

Part 2: Grant API Permissions

Next, you must grant the application the specific permissions required to read user profiles and their mailboxes.

  1. From your new application's page, select API permissions from the left-hand menu.
  2. Click the + Add a permission button.
  3. In the "Request API permissions" pane, select Microsoft Graph.
  4. Select Application permissions. This is critical as it allows the service to run in the background without a user being signed in.
  5. In the "Select permissions" search box, find and check the boxes for the following two permissions:
    • Mail.Read
    • User.Read.All
  6. Click the Add permissions button at the bottom.
  7. Crucial Final Step: You will now see the permissions in your list with a warning status. You must grant consent on behalf of your organization. Click the "Grant admin consent for [Your Organization's Name]" button located above the permissions table. Click Yes in the confirmation dialog. The status for both permissions should now show a green checkmark.

Part 3: Create a Client Secret

The client secret is a password that the archiving service will use to authenticate. Treat this with the same level of security as an administrator's password.

  1. In your application's menu, navigate to Certificates & secrets.
  2. Select the Client secrets tab and click + New client secret.
  3. In the pane that appears:
    • Description: Enter a clear description, such as OpenArchiver Key.
    • Expires: Select an expiry duration. We recommend 12 or 24 months. Set a calendar reminder to renew it before it expires to prevent service interruption.
  4. Click Add.
  5. IMMEDIATELY COPY THE SECRET: The secret is now visible in the "Value" column. This is the only time it will be fully displayed. Copy this value now and store it in a secure password manager before navigating away. If you lose it, you must create a new one.

Part 4: Connecting in OpenArchiver

You now have the three pieces of information required to configure the connection.

  1. Navigate to Ingestion Sources: In the OpenArchiver application, go to the Ingestion Sources page.

  2. Create a New Source: Click the "Create New" button.

  3. Fill in the Configuration Details:

    • Name: Give the source a name (e.g., "Microsoft 365 Archive").
    • Provider: Select "Microsoft 365" from the dropdown.
    • Application (Client) ID: Go to the Overview page of your app registration in the Entra admin center and copy this value.
    • Directory (Tenant) ID: This value is also on the Overview page.
    • Client Secret Value: Paste the secret Value (not the Secret ID) that you copied and saved in the previous step.
  4. Save Changes: Click "Save changes".

What Happens Next?

Once the connection is saved, the system will begin the archiving process:

  1. User Discovery: The service will connect to the Microsoft Graph API to get a list of all users in your organization.
  2. Initial Import: The system will begin a background job to import the mailboxes of all discovered users, folder by folder. The status will show as "Importing". This can take a significant amount of time.
  3. Continuous Sync: After the initial import, the status will change to "Active". The system will use Microsoft Graph's delta query feature to efficiently fetch only new or changed emails, ensuring the archive stays up-to-date.
- +
Skip to content

Connecting to Microsoft 365

This guide provides instructions for Microsoft 365 administrators to set up a connection that allows the archiving of all user mailboxes within their organization.

The connection uses the Microsoft Graph API and an App Registration in Microsoft Entra ID. This is a secure, standard method that grants the archiving service permission to read email data on your behalf without ever needing to handle user passwords.

Prerequisites

  • You must have one of the following administrator roles in your Microsoft 365 tenant: Global Administrator, Application Administrator, or Cloud Application Administrator.

Setup Overview

The setup process involves four main parts, all performed within the Microsoft Entra admin center and the OpenArchiver application:

  1. Registering a new application identity for the archiver in Entra ID.
  2. Granting the application the specific permissions it needs to read mail.
  3. Creating a secure password (a client secret) for the application.
  4. Entering the generated credentials into the OpenArchiver application.

Part 1: Register a New Application in Microsoft Entra ID

First, you will create an "App registration," which acts as an identity for the archiving service within your Microsoft 365 ecosystem.

  1. Sign in to the Microsoft Entra admin center.
  2. In the left-hand navigation pane, go to Identity > Applications > App registrations.
  3. Click the + New registration button at the top of the page.
  4. On the "Register an application" screen:
    • Name: Give the application a descriptive name you will recognize, such as OpenArchiver Service.
    • Supported account types: Select "Accounts in this organizational directory only (Default Directory only - Single tenant)". This is the most secure option.
    • Redirect URI (optional): You can leave this blank.
  5. Click the Register button. You will be taken to the application's main "Overview" page.

Part 2: Grant API Permissions

Next, you must grant the application the specific permissions required to read user profiles and their mailboxes.

  1. From your new application's page, select API permissions from the left-hand menu.
  2. Click the + Add a permission button.
  3. In the "Request API permissions" pane, select Microsoft Graph.
  4. Select Application permissions. This is critical as it allows the service to run in the background without a user being signed in.
  5. In the "Select permissions" search box, find and check the boxes for the following two permissions:
    • Mail.Read
    • User.Read.All
  6. Click the Add permissions button at the bottom.
  7. Crucial Final Step: You will now see the permissions in your list with a warning status. You must grant consent on behalf of your organization. Click the "Grant admin consent for [Your Organization's Name]" button located above the permissions table. Click Yes in the confirmation dialog. The status for both permissions should now show a green checkmark.

Part 3: Create a Client Secret

The client secret is a password that the archiving service will use to authenticate. Treat this with the same level of security as an administrator's password.

  1. In your application's menu, navigate to Certificates & secrets.
  2. Select the Client secrets tab and click + New client secret.
  3. In the pane that appears:
    • Description: Enter a clear description, such as OpenArchiver Key.
    • Expires: Select an expiry duration. We recommend 12 or 24 months. Set a calendar reminder to renew it before it expires to prevent service interruption.
  4. Click Add.
  5. IMMEDIATELY COPY THE SECRET: The secret is now visible in the "Value" column. This is the only time it will be fully displayed. Copy this value now and store it in a secure password manager before navigating away. If you lose it, you must create a new one.

Part 4: Connecting in OpenArchiver

You now have the three pieces of information required to configure the connection.

  1. Navigate to Ingestion Sources: In the OpenArchiver application, go to the Ingestion Sources page.

  2. Create a New Source: Click the "Create New" button.

  3. Fill in the Configuration Details:

    • Name: Give the source a name (e.g., "Microsoft 365 Archive").
    • Provider: Select "Microsoft 365" from the dropdown.
    • Application (Client) ID: Go to the Overview page of your app registration in the Entra admin center and copy this value.
    • Directory (Tenant) ID: This value is also on the Overview page.
    • Client Secret Value: Paste the secret Value (not the Secret ID) that you copied and saved in the previous step.
  4. Save Changes: Click "Save changes".

What Happens Next?

Once the connection is saved, the system will begin the archiving process:

  1. User Discovery: The service will connect to the Microsoft Graph API to get a list of all users in your organization.
  2. Initial Import: The system will begin a background job to import the mailboxes of all discovered users, folder by folder. The status will show as "Importing". This can take a significant amount of time.
  3. Continuous Sync: After the initial import, the status will change to "Active". The system will use Microsoft Graph's delta query feature to efficiently fetch only new or changed emails, ensuring the archive stays up-to-date.
+ \ No newline at end of file diff --git a/user-guides/installation.html b/user-guides/installation.html index 232bfdf..3c5e61a 100644 --- a/user-guides/installation.html +++ b/user-guides/installation.html @@ -3,8 +3,8 @@ - Installation Guide | OpenArchiver Documentation - + Installation Guide | Open Archiver Documentation + @@ -18,7 +18,7 @@ -
Skip to content

Installation Guide

This guide will walk you through setting up Open Archiver using Docker Compose. This is the recommended method for deploying the application.

Prerequisites

  • Docker and Docker Compose installed on your server or local machine.
  • A server or local machine with at least 2GB of RAM.
  • Git installed on your server or local machine.

1. Clone the Repository

First, clone the Open Archiver repository to your machine:

bash
git clone https://github.com/LogicLabs-OU/OpenArchiver.git
+    
Skip to content

Installation Guide

This guide will walk you through setting up Open Archiver using Docker Compose. This is the recommended method for deploying the application.

Prerequisites

  • Docker and Docker Compose installed on your server or local machine.
  • A server or local machine with at least 2GB of RAM.
  • Git installed on your server or local machine.

1. Clone the Repository

First, clone the Open Archiver repository to your machine:

bash
git clone https://github.com/LogicLabs-OU/OpenArchiver.git
 cd OpenArchiver

2. Configure Your Environment

The application is configured using environment variables. You'll need to create a .env file to store your configuration.

Copy the example environment file for Docker:

bash
cp .env.example.docker .env

Now, open the .env file in a text editor and customize the settings.

Important Configuration

You must change the following placeholder values to secure your instance:

  • POSTGRES_PASSWORD: A strong, unique password for the database.
  • REDIS_PASSWORD: A strong, unique password for the Valkey/Redis service.
  • MEILI_MASTER_KEY: A complex key for Meilisearch.
  • JWT_SECRET: A long, random string for signing authentication tokens.
  • ADMIN_PASSWORD: A strong password for the initial admin user.
  • ENCRYPTION_KEY: A 32-byte hex string for encrypting sensitive data in the database. You can generate one with the following command:
    bash
    openssl rand -hex 32

Storage Configuration

By default, the Docker Compose setup uses local filesystem storage, which is persisted using a Docker volume named archiver-data. This is suitable for most use cases.

If you want to use S3-compatible object storage, change the STORAGE_TYPE to s3 and fill in your S3 credentials (STORAGE_S3_* variables).

Using External Services

For convenience, the docker-compose.yml file includes services for PostgreSQL, Valkey (Redis), and Meilisearch. However, you can use your own external or managed instances for these services.

To do so:

  1. Update your .env file: Change the host, port, and credential variables to point to your external service instances. For example, you would update DATABASE_URL, REDIS_HOST, and MEILI_HOST.
  2. Modify docker-compose.yml: Remove or comment out the service definitions for postgres, valkey, and meilisearch from your docker-compose.yml file.

This will configure the Open Archiver application to connect to your services instead of starting the default ones.

Environment Variable Reference

Here is a complete list of environment variables available for configuration:

Application Settings

VariableDescriptionDefault Value
NODE_ENVThe application environment.development
PORT_BACKENDThe port for the backend service.4000
PORT_FRONTENDThe port for the frontend service.3000

Docker Compose Service Configuration

These variables are used by docker-compose.yml to configure the services.

VariableDescriptionDefault Value
POSTGRES_DBThe name of the PostgreSQL database.open_archive
POSTGRES_USERThe username for the PostgreSQL database.admin
POSTGRES_PASSWORDThe password for the PostgreSQL database.password
DATABASE_URLThe connection URL for the PostgreSQL database.postgresql://admin:password@postgres:5432/open_archive
MEILI_MASTER_KEYThe master key for Meilisearch.aSampleMasterKey
MEILI_HOSTThe host for the Meilisearch service.http://meilisearch:7700
REDIS_HOSTThe host for the Valkey (Redis) service.valkey
REDIS_PORTThe port for the Valkey (Redis) service.6379
REDIS_PASSWORDThe password for the Valkey (Redis) service.defaultredispassword
REDIS_TLS_ENABLEDEnable or disable TLS for Redis.false

Storage Settings

VariableDescriptionDefault Value
STORAGE_TYPEThe storage backend to use (local or s3).local
STORAGE_LOCAL_ROOT_PATHThe root path for local file storage./var/data/open-archiver
STORAGE_S3_ENDPOINTThe endpoint for S3-compatible storage.
STORAGE_S3_BUCKETThe bucket name for S3-compatible storage.
STORAGE_S3_ACCESS_KEY_IDThe access key ID for S3-compatible storage.
STORAGE_S3_SECRET_ACCESS_KEYThe secret access key for S3-compatible storage.
STORAGE_S3_REGIONThe region for S3-compatible storage.
STORAGE_S3_FORCE_PATH_STYLEForce path-style addressing for S3.false

Security & Authentication

VariableDescriptionDefault Value
JWT_SECRETA secret key for signing JWT tokens.a-very-secret-key-that-you-should-change
JWT_EXPIRES_INThe expiration time for JWT tokens.7d
ADMIN_EMAILThe email for the initial admin user.admin@local.com
ADMIN_PASSWORDThe password for the initial admin user.a_strong_password_that_you_should_change
SUPER_API_KEYAn API key with super admin privileges.
ENCRYPTION_KEYA 32-byte hex string for encrypting sensitive data.

3. Run the Application

Once you have configured your .env file, you can start all the services using Docker Compose:

bash
docker compose up -d

This command will:

  • Pull the required Docker images for the frontend, backend, database, and other services.
  • Create and start the containers in the background (-d flag).
  • Create the persistent volumes for your data.

You can check the status of the running containers with:

bash
docker compose ps

4. Access the Application

Once the services are running, you can access the Open Archiver web interface by navigating to http://localhost:3000 in your web browser.

You can log in with the ADMIN_EMAIL and ADMIN_PASSWORD you configured in your .env file.

5. Next Steps

After successfully deploying and logging into Open Archiver, the next step is to configure your ingestion sources to start archiving emails.

Updating Your Installation

To update your Open Archiver instance to the latest version, run the following commands:

bash
# Pull the latest changes from the repository
 git pull
 
@@ -27,7 +27,7 @@
 
 # Restart the services with the new images
 docker compose up -d
- + \ No newline at end of file