github/OpenArchiver

Fork 0

mirror of https://github.com/LogicLabs-OU/OpenArchiver.git synced 2026-04-06 00:31:57 +02:00

Files

Wei S. 6e1ebbbfd7 v0.4 init: File encryption, integrity report, deletion protection, job monitoring (#187 )

* open-core setup, adding enterprise package

* enterprise: Audit log API, UI

* Audit-log docs

* feat: Integrity report, allowing users to verify the integrity of archived emails and their attachments.

- When an email is archived, Open Archiver calculates a unique cryptographic signature (a SHA256 hash) for the email's raw `.eml` file and for each of its attachments. These signatures are stored in the database alongside the email's metadata.
- The integrity check feature recalculates these signatures for the stored files and compares them to the original signatures stored in the database. This process allows you to verify that the content of your archived emails has not been altered, corrupted, or tampered with since the moment they were archived.
- Add docs of Integrity report

* Update Docker-compose.yml to use bind mount for Open Archiver data.
Fix API rate-limiter warning about trust proxy

* File encryption support

* Scope attachment deduplication to ingestion source

Previously, attachment deduplication was handled globally by enforcing a unique constraint on the content hash (contentHashSha256) in the `attachments` table. This caused an issue where an attachment from one ingestion source would be incorrectly linked if the same attachment was processed by a different source.

This commit refactors the deduplication logic to be scoped on a per-ingestion-source basis.

Changes:
- **Schema:** The `attachments` table schema has been updated to include a nullable `ingestionSourceId` column. A composite unique index has been added on `(ingestionSourceId, contentHashSha256)` to enforce per-source uniqueness. The `ingestionSourceId` is nullable to ensure backward compatibility with existing databases.
- **Ingestion Logic:** The `IngestionService` has been updated to provide the `ingestionSourceId` when inserting attachment records. The `onConflictDoUpdate` clause now targets the new composite key, ensuring that attachments are only considered duplicates if they have the same hash and originate from the same ingestion source.

* Scope attachment deduplication to ingestion source

This commit refactors the deduplication logic to be scoped on a per-ingestion-source basis.

* Add option to disable deletions

This commit introduces a new feature that allows admins to disable the deletion of emails and ingestion sources for the entire instance. This is a critical feature for compliance and data retention, as it prevents accidental or unauthorized deletions.

Changes:
- **Configuration**: Added an `ENABLE_DELETION` environment variable. If this variable is not set to `true`, all deletion operations will be disabled.
- **Deletion Guard**: A centralized `checkDeletionEnabled` guard has been implemented to enforce this setting at both the controller and service levels, ensuring a robust and secure implementation.
- **Documentation**: The installation guide has been updated to include the new `ENABLE_DELETION` environment variable and its behavior.
- **Refactor**: The `IngestionService`'s `create` method was refactored to remove unnecessary calls to the `delete` method, simplifying the code and improving its robustness.

* Adding position for menu items

* feat(docker): Fix CORS errors

This commit fixes CORS errors when running the app in Docker by introducing the `APP_URL` environment variable. A CORS policy is set up for the backend to only allow origin from the `APP_URL`.

Key changes include:
- New `APP_URL` and `ORIGIN` environment variables have been added to properly configure CORS and the SvelteKit adapter, making the application's public URL easily configurable.
- Dockerfiles are updated to copy the entrypoint script, Drizzle config, and migration files into the final image.
- Documentation and example files (`.env.example`, `docker-compose.yml`) have been updated to reflect these changes.

* feat(attachments): De-duplicate attachment content by content hash

This commit refactors attachment handling to allow multiple emails within the same ingestion source to reference attachments with identical content (same hash).

Changes:
- The unique index on the `attachments` table has been changed to a non-unique index to permit duplicate hash/source pairs.
- The ingestion logic is updated to first check for an existing attachment with the same hash and source. If found, it reuses the existing record; otherwise, it creates a new one. This maintains storage de-duplication.
- The email deletion logic is improved to be more robust. It now correctly removes the email-attachment link before checking if the attachment record and its corresponding file can be safely deleted.

* Not filtering our Trash folder

* feat(backend): Add BullMQ dashboard for job monitoring

This commit introduces a web-based UI for monitoring and managing background jobs using Bullmq.

Key changes:
- A new `/api/v1/jobs` endpoint is created, serving the Bull Board dashboard. Access is restricted to authenticated administrators.
- All BullMQ queue definitions (`ingestion`, `indexing`, `sync-scheduler`) have been centralized into a new `packages/backend/src/jobs/queues.ts` file.
- Workers and services now import queue instances from this central file, improving code organization and removing redundant queue instantiations.

* Add `ALL_INCLUSIVE_ARCHIVE` environment variable to disable jun filtering

* Using BSL license

* frontend: Responsive design for menu bar, pagination

* License service/module

* Remove demoMode logic

* Formatting code

* Remove enterprise packages

* Fix package.json in packages

* Search page responsive fix

---------

Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>

2025-10-24 17:11:05 +02:00

2.8 KiB

Raw Blame History

Troubleshooting CORS Errors

Cross-Origin Resource Sharing (CORS) is a security feature that controls how web applications in one domain can request and interact with resources in another. If not configured correctly, you may encounter errors when performing actions like uploading files.

This guide will help you diagnose and resolve common CORS-related issues.

Symptoms

You may be experiencing a CORS issue if you see one of the following errors in your browser's developer console or in the application's logs:

TypeError: fetch failed
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource.
Unexpected token 'C', "Cross-site"... is not valid JSON

A JSON error response similar to the following:

{
	"message": "CORS Error: This origin is not allowed.",
	"requiredOrigin": "http://localhost:3000",
	"receivedOrigin": "https://localhost:3000"
}

Root Cause

These errors typically occur when the URL you are using to access the application in your browser does not exactly match the APP_URL configured in your .env file.

This can happen for several reasons:

You are accessing the application via a different port.
You are using a reverse proxy that changes the protocol (e.g., from http to https).
The SvelteKit server, in a production build, is incorrectly guessing its public-facing URL.

Solution

The solution is to ensure that the application's frontend and backend are correctly configured with the public-facing URL of your instance. This is done by setting two environment variables: APP_URL and ORIGIN.

Open your .env file in a text editor.
Set APP_URL: Define the APP_URL variable with the exact URL you use to access the application in your browser.
```
APP_URL=http://your-domain-or-ip:3000
```
Set ORIGIN: The SvelteKit server requires a specific ORIGIN variable to correctly identify itself. This should always be set to the value of your APP_URL.
```
ORIGIN=$APP_URL
```
By using $APP_URL, you ensure that both variables are always in sync.

Example Configuration

If you are running the application locally on port 3000, your configuration should look like this:

APP_URL=http://localhost:3000
ORIGIN=$APP_URL

If your application is behind a reverse proxy and is accessible at https://archive.mycompany.com, your configuration should be:

APP_URL=https://archive.mycompany.com
ORIGIN=$APP_URL

After making these changes to your .env file, you must restart the application for them to take effect:

docker compose up -d --force-recreate

This will ensure that the backend's CORS policy and the frontend server's origin are correctly aligned, resolving the errors.

2.8 KiB Raw Blame History