Increase file upload limit and improve ingestion robustness #302

New Issue

MrUnknownDE · 2026-04-05T16:17:20+02:00

MrUnknownDE commented

2026-04-05 16:17:20 +02:00

Originally created by @wayneshn on 8/13/2025

This pull request introduces two primary enhancements to improve the usability and reliability of the data ingestion process. It addresses failures related to sender-less emails in PST files and removes the bottleneck for uploading large archive files.

Key Changes:

Fix for Sender-less Emails in PST/EML Files
- The ingestion process would fail when encountering an email within a .pst or .eml file that was missing a From address.
- Both the PSTConnector and EMLConnector have been updated to gracefully handle this edge case. If an email's from field is empty, it is now assigned a default value of 'No Sender'. This prevents the ingestion worker from crashing and ensures that all messages are successfully archived.
Configurable File Upload Limit for Ingestion
- Previously, the frontend server had a restrictive default body size limit, preventing administrators from uploading large .pst files for ingestion.
- A new environment variable, FRONTEND_BODY_SIZE_LIMIT, has been added to .env.example with a default of 100M. V0.4.2 (#32)
- This variable is now used in the SvelteKit adapter configuration (svelte.config.js) to control the maximum request body size. This allows system administrators to easily adjust the limit based on their needs, facilitating the import of large historical archives.
Improved Frontend Error Reporting
- To aid in debugging, the error handling on the IngestionSourceForm component has been improved.
- Instead of displaying a generic "Upload Failed" message, the UI will now show the specific error message returned from the backend API. This provides immediate and actionable feedback to the administrator if an upload fails, for instance, by exceeding the newly configured FRONTEND_BODY_SIZE_LIMIT.

*Originally created by @wayneshn on 8/13/2025* This pull request introduces two primary enhancements to improve the usability and reliability of the data ingestion process. It addresses failures related to sender-less emails in PST files and removes the bottleneck for uploading large archive files. #### Key Changes: * **Fix for Sender-less Emails in PST/EML Files** * The ingestion process would fail when encountering an email within a `.pst` or `.eml` file that was missing a `From` address. * Both the `PSTConnector` and `EMLConnector` have been updated to gracefully handle this edge case. If an email's `from` field is empty, it is now assigned a default value of `'No Sender'`. This prevents the ingestion worker from crashing and ensures that all messages are successfully archived. * **Configurable File Upload Limit for Ingestion** * Previously, the frontend server had a restrictive default body size limit, preventing administrators from uploading large `.pst` files for ingestion. * A new environment variable, `FRONTEND_BODY_SIZE_LIMIT`, has been added to `.env.example` with a default of `100M`. #32 * This variable is now used in the SvelteKit adapter configuration (`svelte.config.js`) to control the maximum request body size. This allows system administrators to easily adjust the limit based on their needs, facilitating the import of large historical archives. * **Improved Frontend Error Reporting** * To aid in debugging, the error handling on the `IngestionSourceForm` component has been improved. * Instead of displaying a generic "Upload Failed" message, the UI will now show the specific error message returned from the backend API. This provides immediate and actionable feedback to the administrator if an upload fails, for instance, by exceeding the newly configured `FRONTEND_BODY_SIZE_LIMIT`.

MrUnknownDE closed this issue

2026-04-05 16:17:20 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/OpenArchiver#302