v0.4 init: File encryption, integrity report, deletion protection, job monitoring (#187)

* open-core setup, adding enterprise package

* enterprise: Audit log API, UI

* Audit-log docs

* feat: Integrity report, allowing users to verify the integrity of archived emails and their attachments.

- When an email is archived, Open Archiver calculates a unique cryptographic signature (a SHA256 hash) for the email's raw `.eml` file and for each of its attachments. These signatures are stored in the database alongside the email's metadata.
- The integrity check feature recalculates these signatures for the stored files and compares them to the original signatures stored in the database. This process allows you to verify that the content of your archived emails has not been altered, corrupted, or tampered with since the moment they were archived.
- Add docs of Integrity report

* Update Docker-compose.yml to use bind mount for Open Archiver data.
Fix API rate-limiter warning about trust proxy

* File encryption support

* Scope attachment deduplication to ingestion source

Previously, attachment deduplication was handled globally by enforcing a unique constraint on the content hash (contentHashSha256) in the `attachments` table. This caused an issue where an attachment from one ingestion source would be incorrectly linked if the same attachment was processed by a different source.

This commit refactors the deduplication logic to be scoped on a per-ingestion-source basis.

Changes:
-   **Schema:** The `attachments` table schema has been updated to include a nullable `ingestionSourceId` column. A composite unique index has been added on `(ingestionSourceId, contentHashSha256)` to enforce per-source uniqueness. The `ingestionSourceId` is nullable to ensure backward compatibility with existing databases.
-   **Ingestion Logic:** The `IngestionService` has been updated to provide the `ingestionSourceId` when inserting attachment records. The `onConflictDoUpdate` clause now targets the new composite key, ensuring that attachments are only considered duplicates if they have the same hash and originate from the same ingestion source.

* Scope attachment deduplication to ingestion source

Previously, attachment deduplication was handled globally by enforcing a unique constraint on the content hash (contentHashSha256) in the `attachments` table. This caused an issue where an attachment from one ingestion source would be incorrectly linked if the same attachment was processed by a different source.

This commit refactors the deduplication logic to be scoped on a per-ingestion-source basis.

Changes:
-   **Schema:** The `attachments` table schema has been updated to include a nullable `ingestionSourceId` column. A composite unique index has been added on `(ingestionSourceId, contentHashSha256)` to enforce per-source uniqueness. The `ingestionSourceId` is nullable to ensure backward compatibility with existing databases.
-   **Ingestion Logic:** The `IngestionService` has been updated to provide the `ingestionSourceId` when inserting attachment records. The `onConflictDoUpdate` clause now targets the new composite key, ensuring that attachments are only considered duplicates if they have the same hash and originate from the same ingestion source.

* Add option to disable deletions

This commit introduces a new feature that allows admins to disable the deletion of emails and ingestion sources for the entire instance. This is a critical feature for compliance and data retention, as it prevents accidental or unauthorized deletions.

Changes:
-   **Configuration**: Added an `ENABLE_DELETION` environment variable. If this variable is not set to `true`, all deletion operations will be disabled.
-   **Deletion Guard**: A centralized `checkDeletionEnabled` guard has been implemented to enforce this setting at both the controller and service levels, ensuring a robust and secure implementation.
-   **Documentation**: The installation guide has been updated to include the new `ENABLE_DELETION` environment variable and its behavior.
-   **Refactor**: The `IngestionService`'s `create` method was refactored to remove unnecessary calls to the `delete` method, simplifying the code and improving its robustness.

* Adding position for menu items

* feat(docker): Fix CORS errors

This commit fixes CORS errors when running the app in Docker by introducing the `APP_URL` environment variable. A CORS policy is set up for the backend to only allow origin from the `APP_URL`.

Key changes include:
- New `APP_URL` and `ORIGIN` environment variables have been added to properly configure CORS and the SvelteKit adapter, making the application's public URL easily configurable.
- Dockerfiles are updated to copy the entrypoint script, Drizzle config, and migration files into the final image.
- Documentation and example files (`.env.example`, `docker-compose.yml`) have been updated to reflect these changes.

* feat(attachments): De-duplicate attachment content by content hash

This commit refactors attachment handling to allow multiple emails within the same ingestion source to reference attachments with identical content (same hash).

Changes:
- The unique index on the `attachments` table has been changed to a non-unique index to permit duplicate hash/source pairs.
- The ingestion logic is updated to first check for an existing attachment with the same hash and source. If found, it reuses the existing record; otherwise, it creates a new one. This maintains storage de-duplication.
- The email deletion logic is improved to be more robust. It now correctly removes the email-attachment link before checking if the attachment record and its corresponding file can be safely deleted.

* Not filtering our Trash folder

* feat(backend): Add BullMQ dashboard for job monitoring

This commit introduces a web-based UI for monitoring and managing background jobs using Bullmq.

Key changes:
- A new `/api/v1/jobs` endpoint is created, serving the Bull Board dashboard. Access is restricted to authenticated administrators.
- All BullMQ queue definitions (`ingestion`, `indexing`, `sync-scheduler`) have been centralized into a new `packages/backend/src/jobs/queues.ts` file.
- Workers and services now import queue instances from this central file, improving code organization and removing redundant queue instantiations.

* Add `ALL_INCLUSIVE_ARCHIVE` environment variable to disable jun filtering

* Using BSL license

* frontend: Responsive design for menu bar, pagination

* License service/module

* Remove demoMode logic

* Formatting code

* Remove enterprise packages

* Fix package.json in packages

* Search page responsive fix

---------

Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>

.env.example

@@ -4,8 +4,15 @@
 NODE_ENV=development
 PORT_BACKEND=4000
 PORT_FRONTEND=3000
+# The public-facing URL of your application. This is used by the backend to configure CORS.
+APP_URL=http://localhost:3000
+# This is used by the SvelteKit Node adapter to determine the server's public-facing URL.
+# It should always be set to the value of APP_URL.
+ORIGIN=$APP_URL
 # The frequency of continuous email syncing. Default is every minutes, but you can change it to another value based on your needs.
 SYNC_FREQUENCY='* * * * *' 
+# Set to 'true' to include Junk and Trash folders in the email archive. Defaults to false.
+ALL_INCLUSIVE_ARCHIVE=false
 
 # --- Docker Compose Service Configuration ---
 # These variables are used by docker-compose.yml to configure the services. Leave them unchanged if you use Docker services for Postgresql, Valkey (Redis) and Meilisearch. If you decide to use your own instances of these services, you can substitute them with your own connection credentials.
@@ -40,7 +47,9 @@ BODY_SIZE_LIMIT=100M
 # --- Local Storage Settings ---
 # The path inside the container where files will be stored.
 # This is mapped to a Docker volume for persistence.
-# This is only used if STORAGE_TYPE is 'local'.
+# This is not an optional variable, it is where the Open Archiver service stores application data. Set this even if you are using S3 storage.
+# Make sure the user that runs the Open Archiver service has read and write access to this path.
+# Important: It is recommended to create this path manually before installation, otherwise you may face permission and ownership problems.
 STORAGE_LOCAL_ROOT_PATH=/var/data/open-archiver
 
 # --- S3-Compatible Storage Settings ---
@@ -53,8 +62,18 @@ STORAGE_S3_REGION=
 # Set to 'true' for MinIO and other non-AWS S3 services
 STORAGE_S3_FORCE_PATH_STYLE=false
 
+# --- Storage Encryption ---
+# IMPORTANT: Generate a secure, random 32-byte hex string for this key.
+# You can use `openssl rand -hex 32` to generate a key.
+# This key is used for AES-256 encryption of files at rest.
+# This is an optional variable, if not set, files will not be encrypted.
+STORAGE_ENCRYPTION_KEY=
+
 # --- Security & Authentication ---
 
+# Enable or disable deletion of emails and ingestion sources. Defaults to false.
+ENABLE_DELETION=false
+
 # Rate Limiting
 # The window in milliseconds for which API requests are checked. Defaults to 60000 (1 minute).
 RATE_LIMIT_WINDOW_MS=60000
@@ -77,5 +96,3 @@ ENCRYPTION_KEY=
 # Apache Tika Integration
 # ONLY active if TIKA_URL is set
 TIKA_URL=http://tika:9998
-
-