v0.4 init: File encryption, integrity report, deletion protection, job monitoring #148

New Issue

MrUnknownDE · 2026-04-05T16:16:45+02:00

MrUnknownDE commented

2026-04-05 16:16:45 +02:00

Originally created by @wayneshn on 10/17/2025

This pull request introduces a set of features focused on enhancing security, compliance, and administration capabilities. It also includes important refactors to core systems like attachment deduplication and background job management.

Major Features & Enhancements

Compliance:

Audit Logging: Implements the API and UI for a comprehensive audit trail, allowing administrators to track user logins, searches, exports, and policy changes. This is a critical feature for compliance.
Deletion Guard: A new instance-wide setting (ENABLE_DELETION) allows administrators to disable the deletion of emails and ingestion sources, providing a crucial safeguard for compliance and preventing accidental data loss.
Data Integrity Reports: A new verification feature has been added. The system now calculates and stores a SHA256 hash for every archived email and attachment. The integrity report recalculates these signatures on demand to verify that archived data has not been altered or corrupted.

Security:

File Encryption at Rest: All archived files can now be encrypted, adding a critical layer of security that directly fulfills a core product requirement.

System Administration:

Job Monitoring Dashboard: A BullMQ dashboard has been integrated and is now available to administrators at the /api/v1/jobs endpoint. This provides a web UI to monitor the status of all background job queues (e.g., ingestion, indexing), enhancing system observability for administrators.
Centralized Queues: As part of the dashboard integration, all BullMQ queue definitions have been refactored into a central location (packages/backend/src/jobs/queues.ts) to improve code organization and eliminate redundant instantiations.

Scoped Attachment Deduplication: The attachment deduplication logic has been significantly refactored. It is now scoped to each ingestion source, fixing a critical bug where an attachment from one source could be incorrectly linked to an email from another. This ensures single-instance storage is applied correctly and robustly on a per-source basis.
Docker & Deployment:
- The docker-compose.yml file has been updated to use bind mounts for persistent data.
- Fixed a trust proxy warning with the API rate-limiter.
- Resolved CORS errors encountered in Docker deployments.
Ingestion: The trash folder is no longer filtered out during email synchronization.
UI: Added a position property to menu items for consistent ordering.

*Originally created by @wayneshn on 10/17/2025* This pull request introduces a set of features focused on enhancing security, compliance, and administration capabilities. It also includes important refactors to core systems like attachment deduplication and background job management. ### Major Features & Enhancements **Compliance:** * **Audit Logging:** Implements the API and UI for a comprehensive audit trail, allowing administrators to track user logins, searches, exports, and policy changes. This is a critical feature for compliance. * **Deletion Guard:** A new instance-wide setting (`ENABLE_DELETION`) allows administrators to disable the deletion of emails and ingestion sources, providing a crucial safeguard for compliance and preventing accidental data loss. * **Data Integrity Reports:** A new verification feature has been added. The system now calculates and stores a `SHA256` hash for every archived email and attachment. The integrity report recalculates these signatures on demand to verify that archived data has not been altered or corrupted. **Security:** * **File Encryption at Rest:** All archived files can now be encrypted, adding a critical layer of security that directly fulfills a core product requirement. **System Administration:** * **Job Monitoring Dashboard:** A BullMQ dashboard has been integrated and is now available to administrators at the `/api/v1/jobs` endpoint. This provides a web UI to monitor the status of all background job queues (e.g., ingestion, indexing), enhancing system observability for administrators. * **Centralized Queues:** As part of the dashboard integration, all BullMQ queue definitions have been refactored into a central location (`packages/backend/src/jobs/queues.ts`) to improve code organization and eliminate redundant instantiations. ### Refinements and Bug Fixes * **Scoped Attachment Deduplication:** The attachment deduplication logic has been significantly refactored. It is now scoped to each ingestion source, fixing a critical bug where an attachment from one source could be incorrectly linked to an email from another. This ensures single-instance storage is applied correctly and robustly on a per-source basis. * **Docker & Deployment:** * The `docker-compose.yml` file has been updated to use bind mounts for persistent data. * Fixed a `trust proxy` warning with the API rate-limiter. * Resolved CORS errors encountered in Docker deployments. * **Ingestion:** The trash folder is no longer filtered out during email synchronization. * **UI:** Added a position property to menu items for consistent ordering.

MrUnknownDE closed this issue

2026-04-05 16:16:45 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/OpenArchiver#148