* Feat/tika integration (#94)
* feat(Tika) Integration von Tika zur Textextraktion
* feat(Tika) Integration of Apache Tika for text extraction
* feat(Tika): Complete Tika integration with text extraction and docker-compose setup
- Add Tika service to docker-compose.yml
- Implement text sanitization and document validation
- Improve batch processing with concurrency control
* fix(comments) translated comments into english
fix(docker) removed ports (only used for testing)
* feat(indexing): Implement batch indexing for Meilisearch
This change introduces batch processing for indexing emails into Meilisearch to significantly improve performance and throughput during ingestion. This change is based on the batch processing method previously contributed by @axeldunkel.
Previously, each email was indexed individually, resulting in a high number of separate API calls. This approach was inefficient, especially for large mailboxes.
The `processMailbox` queue worker now accumulates emails into a batch before sending them to the `IndexingService`. The service then uses the `addDocuments` Meilisearch API endpoint to index the entire batch in a single request, reducing network overhead and improving indexing speed.
A new environment variable, `MEILI_INDEXING_BATCH`, has been added to make the batch size configurable, with a default of 500.
Additionally, this commit includes minor refactoring:
- The `TikaService` has been moved to its own dedicated file.
- The `PendingEmail` type has been moved to the shared `@open-archiver/types` package.
* chore(jobs): make continuous sync job scheduling idempotent
Adds a static `jobId` to the repeatable 'schedule-continuous-sync' job.
This prevents duplicate jobs from being scheduled if the server restarts. By providing a unique ID, the queue will update the existing repeatable job instead of creating a new one, ensuring the sync runs only at the configured frequency.
---------
Co-authored-by: axeldunkel <53174090+axeldunkel@users.noreply.github.com>
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
This commit introduces two major features:
1. **Mbox File Ingestion:**
Users can now ingest emails from Mbox files (`.mbox`). A new Mbox connector has been implemented on the backend, and the user interface has been updated to support creating Mbox ingestion sources. Documentation for this new provider has also been added.
Additionally, this commit includes new documentation for upgrading and migrating Open Archiver.
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
This commit updates several dependencies in the frontend and backend packages.
- **Backend:**
- Upgrades `xlsx` to version `0.20.3` by pointing to the official CDN URL. This ensures usage of the community edition with a permissive license.
- Removes the unused `bull-board` development dependency.
- **Frontend:**
- Upgrades `@sveltejs/kit` from `^2.16.0` to `^2.38.1` to stay current with the latest features and fixes.
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* feat: Add new version notification in footer
This commit implements a system to check for new application versions and notify the user.
On page load, the server-side code now fetches the latest release from the GitHub repository API. It uses `semver` to compare the current application version with the latest release tag.
If a newer version is available, an alert is displayed in the footer with a link to the release page. The current application version is also now displayed in the footer. The version check is cached for one hour to minimize API requests.
* Modify version notification
* current version 0.3.1
* Resolve conflicts
* Code formatting
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
This commit implements a system to check for new application versions and notify the user.
On page load, the server-side code now fetches the latest release from the GitHub repository API. It uses `semver` to compare the current application version with the latest release tag.
If a newer version is available, an alert is displayed in the footer with a link to the release page. The current application version is also now displayed in the footer. The version check is cached for one hour to minimize API requests.
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* feat(auth): Implement API key authentication
This commit enables API access with an API key system. This change provides a better experience for programmatic access and third-party integrations.
Key changes include:
- **API Key Management:** Users can now generate, manage, and revoke persistent API keys through a new "API Keys" section in the settings UI.
- **Authentication Middleware:** API requests are now authenticated via an `X-API-KEY` header instead of the previous `Authorization: Bearer` token.
- **Backend Implementation:** Adds a new `api_keys` database table, along with corresponding services, controllers, and routes to manage the key lifecycle securely.
- **Rate Limiting:** The API rate limiter now uses the API key to identify and track requests.
- **Documentation:** The API authentication documentation has been updated to reflect the new method.
* Add configurable API rate limiting
Two new variables are added to `.env.example`:
- `RATE_LIMIT_WINDOW_MS`: The time window in milliseconds for which requests are checked (defaults to 15 minutes).
- `RATE_LIMIT_MAX_REQUESTS`: The maximum number of requests allowed from an IP within the window (defaults to 100).
The installation documentation has been updated to reflect these new configuration options.
* Disable API operation in demo mode
* Exclude public API endpoints from rate limiting
* JSON rate limiting message & status code
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* feat(auth): Implement API key authentication
This commit enables API access with an API key system. This change provides a better experience for programmatic access and third-party integrations.
Key changes include:
- **API Key Management:** Users can now generate, manage, and revoke persistent API keys through a new "API Keys" section in the settings UI.
- **Authentication Middleware:** API requests are now authenticated via an `X-API-KEY` header instead of the previous `Authorization: Bearer` token.
- **Backend Implementation:** Adds a new `api_keys` database table, along with corresponding services, controllers, and routes to manage the key lifecycle securely.
- **Rate Limiting:** The API rate limiter now uses the API key to identify and track requests.
- **Documentation:** The API authentication documentation has been updated to reflect the new method.
* Add configurable API rate limiting
Two new variables are added to `.env.example`:
- `RATE_LIMIT_WINDOW_MS`: The time window in milliseconds for which requests are checked (defaults to 15 minutes).
- `RATE_LIMIT_MAX_REQUESTS`: The maximum number of requests allowed from an IP within the window (defaults to 100).
The installation documentation has been updated to reflect these new configuration options.
* Disable API operation in demo mode
* Exclude public API endpoints from rate limiting
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* feat(auth): Implement API key authentication
This commit enables API access with an API key system. This change provides a better experience for programmatic access and third-party integrations.
Key changes include:
- **API Key Management:** Users can now generate, manage, and revoke persistent API keys through a new "API Keys" section in the settings UI.
- **Authentication Middleware:** API requests are now authenticated via an `X-API-KEY` header instead of the previous `Authorization: Bearer` token.
- **Backend Implementation:** Adds a new `api_keys` database table, along with corresponding services, controllers, and routes to manage the key lifecycle securely.
- **Rate Limiting:** The API rate limiter now uses the API key to identify and track requests.
- **Documentation:** The API authentication documentation has been updated to reflect the new method.
* Add configurable API rate limiting
Two new variables are added to `.env.example`:
- `RATE_LIMIT_WINDOW_MS`: The time window in milliseconds for which requests are checked (defaults to 15 minutes).
- `RATE_LIMIT_MAX_REQUESTS`: The maximum number of requests allowed from an IP within the window (defaults to 100).
The installation documentation has been updated to reflect these new configuration options.
* Disable API operation in demo mode
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* feat(auth): Implement API key authentication
This commit enables API access with an API key system. This change provides a better experience for programmatic access and third-party integrations.
Key changes include:
- **API Key Management:** Users can now generate, manage, and revoke persistent API keys through a new "API Keys" section in the settings UI.
- **Authentication Middleware:** API requests are now authenticated via an `X-API-KEY` header instead of the previous `Authorization: Bearer` token.
- **Backend Implementation:** Adds a new `api_keys` database table, along with corresponding services, controllers, and routes to manage the key lifecycle securely.
- **Rate Limiting:** The API rate limiter now uses the API key to identify and track requests.
- **Documentation:** The API authentication documentation has been updated to reflect the new method.
* Add configurable API rate limiting
Two new variables are added to `.env.example`:
- `RATE_LIMIT_WINDOW_MS`: The time window in milliseconds for which requests are checked (defaults to 15 minutes).
- `RATE_LIMIT_MAX_REQUESTS`: The maximum number of requests allowed from an IP within the window (defaults to 100).
The installation documentation has been updated to reflect these new configuration options.
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* Disable system settings for demo mode
* Status API response: needsSetup
* Remove SUPER_API_KEY support
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* System settings setup
* Multi-language support
* feat: Add internationalization (i18n) support to frontend
This commit introduces internationalization (i18n) to the frontend using the `sveltekit-i18n` library, allowing the user interface to be translated into multiple languages.
Key changes:
- Added translation files for 10 languages (en, de, es, fr, etc.).
- Replaced hardcoded text strings throughout the frontend components and pages with translation keys.
- Added a language selector to the system settings page, allowing administrators to set the default application language.
- Updated the backend settings API to store and expose the new language configuration.
* Adding greek translation
* feat(backend): Implement i18n for API responses
This commit introduces internationalization (i18n) to the backend API using the `i18next` library.
Hardcoded error and response messages in the API controllers have been replaced with translation keys, which are processed by the new i18next middleware. This allows for API responses to be translated into different languages.
The following dependencies were added:
- `i18next`
- `i18next-fs-backend`
- `i18next-http-middleware`
* Formatting code
* Translation revamp for frontend and backend, adding systems docs
* Docs site title
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* Format checked, contributing.md update
* Middleware setup
* IAP API, create user/roles in frontend
* RBAC using CASL library
* Switch to CASL, secure search, resource-level access control
* Remove inherent behavior, index userEmail, adding docs for IAM policies
* Format
* System settings setup
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* Format checked, contributing.md update
* Middleware setup
* IAP API, create user/roles in frontend
* RBAC using CASL library
* Switch to CASL, secure search, resource-level access control
* Remove inherent behavior, index userEmail, adding docs for IAM policies
* Format
* Adding IAM policy documentation to Docs site
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* Format checked, contributing.md update
* Middleware setup
* IAP API, create user/roles in frontend
* RBAC using CASL library
* Switch to CASL, secure search, resource-level access control
* Remove inherent behavior, index userEmail, adding docs for IAM policies
* Format
---------
Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
* define base image arg
* create base stage with common content
* chmod executable entrypoint file
this avoids re-copying the same file as is being modified in the docker
layer
* cache npm downloaded packages
avoids re-downloading deps if cache content is available
Initialize newMaxUids with lastUid instead of mailbox maximum
to prevent marking unfetched emails as synced.
The bug sets newMaxUids to the highest UID before fetching,
causing all existing emails to be skipped when sync state
is saved early.
Fixes#45
