V0.4.0 fix (#204)

* Jobs page responsive fix

* feat(ingestion): Refactor email indexing into a dedicated background job

This commit refactors the email indexing process to improve the performance and reliability of the ingestion pipeline.

Previously, email indexing was performed synchronously within the mailbox processing job. This could lead to timeouts and failed ingestion cycles if the indexing step was slow or encountered errors.

To address this, the indexing logic has been moved into a separate, dedicated background job queue (`indexingQueue`). Now, the mailbox processor simply adds a batch of emails to this queue. A separate worker then processes the indexing job asynchronously.

This decoupling makes the ingestion process more robust:
- It prevents slow indexing from blocking or failing the entire mailbox sync.
- It allows for better resource management and scalability by handling indexing in a dedicated process.
- It improves error handling, as a failed indexing job can be retried independently without affecting the main ingestion flow.

Additionally, this commit includes minor documentation updates and removes a premature timeout in the PDF text extraction helper that was causing issues.

---------

Co-authored-by: Wayne <5291640+ringoinca@users.noreply.github.com>
This commit is contained in:
Wei S.
2025-10-28 13:14:43 +01:00
committed by GitHub
parent 6e1ebbbfd7
commit 42b0f6e5f1
15 changed files with 286 additions and 94 deletions

View File

@@ -7,7 +7,7 @@
[![Redis](https://img.shields.io/badge/Redis-DC382D?style=for-the-badge&logo=redis&logoColor=white)](https://redis.io) [![Redis](https://img.shields.io/badge/Redis-DC382D?style=for-the-badge&logo=redis&logoColor=white)](https://redis.io)
[![SvelteKit](https://img.shields.io/badge/SvelteKit-FF3E00?style=for-the-badge&logo=svelte&logoColor=white)](https://svelte.dev/) [![SvelteKit](https://img.shields.io/badge/SvelteKit-FF3E00?style=for-the-badge&logo=svelte&logoColor=white)](https://svelte.dev/)
**A secure, sovereign, and open-source platform for email archiving and eDiscovery.** **A secure, sovereign, and open-source platform for email archiving.**
Open Archiver provides a robust, self-hosted solution for archiving, storing, indexing, and searching emails from major platforms, including Google Workspace (Gmail), Microsoft 365, PST files, as well as generic IMAP-enabled email inboxes. Use Open Archiver to keep a permanent, tamper-proof record of your communication history, free from vendor lock-in. Open Archiver provides a robust, self-hosted solution for archiving, storing, indexing, and searching emails from major platforms, including Google Workspace (Gmail), Microsoft 365, PST files, as well as generic IMAP-enabled email inboxes. Use Open Archiver to keep a permanent, tamper-proof record of your communication history, free from vendor lock-in.
@@ -48,13 +48,13 @@ Password: openarchiver_demo
- Zipped .eml files - Zipped .eml files
- Mbox files - Mbox files
- **Secure & Efficient Storage**: Emails are stored in the standard `.eml` format. The system uses deduplication and compression to minimize storage costs. All data is encrypted at rest. - **Secure & Efficient Storage**: Emails are stored in the standard `.eml` format. The system uses deduplication and compression to minimize storage costs. All files are encrypted at rest.
- **Pluggable Storage Backends**: Support both local filesystem storage and S3-compatible object storage (like AWS S3 or MinIO). - **Pluggable Storage Backends**: Support both local filesystem storage and S3-compatible object storage (like AWS S3 or MinIO).
- **Powerful Search & eDiscovery**: A high-performance search engine indexes the full text of emails and attachments (PDF, DOCX, etc.). - **Powerful Search & eDiscovery**: A high-performance search engine indexes the full text of emails and attachments (PDF, DOCX, etc.).
- **Thread discovery**: The ability to discover if an email belongs to a thread/conversation and present the context. - **Thread discovery**: The ability to discover if an email belongs to a thread/conversation and present the context.
- **Compliance & Retention**: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD). - **Compliance & Retention**: Define granular retention policies to automatically manage the lifecycle of your data. Place legal holds on communications to prevent deletion during litigation (TBD).
- **File Hash and Encryption**: Email and attachment file hash values are stored in the meta database upon ingestion, meaning any attempt to alter the file content will be identified, ensuring legal and regulatory compliance. - **File Hash and Encryption**: Email and attachment file hash values are stored in the meta database upon ingestion, meaning any attempt to alter the file content will be identified, ensuring legal and regulatory compliance.
- **Comprehensive Auditing**: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when (TBD). - **Comprehensive Auditing**: An immutable audit trail logs all system activities, ensuring you have a clear record of who accessed what and when.
## 🛠️ Tech Stack ## 🛠️ Tech Stack

Binary file not shown.

After

Width:  |  Height:  |  Size: 259 KiB

View File

@@ -1,4 +1,4 @@
# Email Integrity Check # Integrity Check
Open Archiver allows you to verify the integrity of your archived emails and their attachments. This guide explains how the integrity check works and what the results mean. Open Archiver allows you to verify the integrity of your archived emails and their attachments. This guide explains how the integrity check works and what the results mean.

View File

@@ -47,10 +47,10 @@ function extractTextFromPdf(buffer: Buffer): Promise<string> {
} }
// reduced Timeout for better performance // reduced Timeout for better performance
setTimeout(() => { // setTimeout(() => {
logger.warn('PDF parsing timed out'); // logger.warn('PDF parsing timed out');
finish(''); // finish('');
}, 5000); // }, 5000);
}); });
} }

View File

@@ -33,7 +33,6 @@ export const processMailboxProcessor = async (job: Job<IProcessMailboxJob, SyncS
const searchService = new SearchService(); const searchService = new SearchService();
const storageService = new StorageService(); const storageService = new StorageService();
const databaseService = new DatabaseService(); const databaseService = new DatabaseService();
const indexingService = new IndexingService(databaseService, searchService, storageService);
try { try {
const source = await IngestionService.findById(ingestionSourceId); const source = await IngestionService.findById(ingestionSourceId);
@@ -72,7 +71,8 @@ export const processMailboxProcessor = async (job: Job<IProcessMailboxJob, SyncS
return newSyncState; return newSyncState;
} catch (error) { } catch (error) {
if (emailBatch.length > 0) { if (emailBatch.length > 0) {
await indexingService.indexEmailBatch(emailBatch); await indexingQueue.add('index-email-batch', { emails: emailBatch });
emailBatch = [];
} }
logger.error({ err: error, ingestionSourceId, userEmail }, 'Error processing mailbox'); logger.error({ err: error, ingestionSourceId, userEmail }, 'Error processing mailbox');

View File

@@ -49,9 +49,10 @@ export default async (job: Job<ISyncCycleFinishedJob, any, string>) => {
// if data doesn't have error property, it is a successful job with SyncState // if data doesn't have error property, it is a successful job with SyncState
const successfulJobs = allChildJobs.filter((v) => !v || !(v as any).error) as SyncState[]; const successfulJobs = allChildJobs.filter((v) => !v || !(v as any).error) as SyncState[];
const finalSyncState = deepmerge( const finalSyncState =
...successfulJobs.filter((s) => s && Object.keys(s).length > 0) successfulJobs.length > 0
); ? deepmerge(...successfulJobs.filter((s) => s && Object.keys(s).length > 0))
: {};
const source = await IngestionService.findById(ingestionSourceId); const source = await IngestionService.findById(ingestionSourceId);
let status: IngestionStatus = 'active'; let status: IngestionStatus = 'active';
@@ -63,7 +64,9 @@ export default async (job: Job<ISyncCycleFinishedJob, any, string>) => {
let message: string; let message: string;
// Check for a specific rate-limit message from the successful jobs // Check for a specific rate-limit message from the successful jobs
const rateLimitMessage = successfulJobs.find((j) => j.statusMessage)?.statusMessage; const rateLimitMessage = successfulJobs.find(
(j) => j.statusMessage && j.statusMessage.includes('rate limit')
)?.statusMessage;
if (failedJobs.length > 0) { if (failedJobs.length > 0) {
status = 'error'; status = 'error';

View File

@@ -93,21 +93,17 @@ export class IndexingService {
const batch = emails.slice(i, i + CONCURRENCY_LIMIT); const batch = emails.slice(i, i + CONCURRENCY_LIMIT);
const batchDocuments = await Promise.allSettled( const batchDocuments = await Promise.allSettled(
batch.map(async ({ email, sourceId, archivedId }) => { batch.map(async (pendingEmail) => {
try { try {
return await this.createEmailDocumentFromRawForBatch( const document = await this.indexEmailById(pendingEmail.archivedEmailId);
email, if (document) {
sourceId, return document;
archivedId, }
email.userEmail || '' return null;
);
} catch (error) { } catch (error) {
logger.error( logger.error(
{ {
emailId: archivedId, emailId: pendingEmail.archivedEmailId,
sourceId,
userEmail: email.userEmail || '',
rawEmailData: JSON.stringify(email, null, 2),
error: error instanceof Error ? error.message : String(error), error: error instanceof Error ? error.message : String(error),
}, },
'Failed to create document for email in batch' 'Failed to create document for email in batch'
@@ -118,10 +114,12 @@ export class IndexingService {
); );
for (const result of batchDocuments) { for (const result of batchDocuments) {
if (result.status === 'fulfilled') { if (result.status === 'fulfilled' && result.value) {
rawDocuments.push(result.value); rawDocuments.push(result.value);
} else { } else if (result.status === 'rejected') {
logger.error({ error: result.reason }, 'Failed to process email in batch'); logger.error({ error: result.reason }, 'Failed to process email in batch');
} else {
logger.error({ result: result }, 'Failed to process email in batch, reason unknown.');
} }
} }
} }
@@ -195,10 +193,7 @@ export class IndexingService {
} }
} }
/** private async indexEmailById(emailId: string): Promise<EmailDocument | null> {
* @deprecated
*/
private async indexEmailById(emailId: string): Promise<void> {
const email = await this.dbService.db.query.archivedEmails.findFirst({ const email = await this.dbService.db.query.archivedEmails.findFirst({
where: eq(archivedEmails.id, emailId), where: eq(archivedEmails.id, emailId),
}); });
@@ -228,13 +223,13 @@ export class IndexingService {
emailAttachmentsResult, emailAttachmentsResult,
email.userEmail email.userEmail
); );
await this.searchService.addDocuments('emails', [document], 'id'); return document;
} }
/** /**
* @deprecated * @deprecated
*/ */
private async indexByEmail(pendingEmail: PendingEmail): Promise<void> { /* private async indexByEmail(pendingEmail: PendingEmail): Promise<void> {
const attachments: AttachmentsType = []; const attachments: AttachmentsType = [];
if (pendingEmail.email.attachments && pendingEmail.email.attachments.length > 0) { if (pendingEmail.email.attachments && pendingEmail.email.attachments.length > 0) {
for (const attachment of pendingEmail.email.attachments) { for (const attachment of pendingEmail.email.attachments) {
@@ -254,12 +249,12 @@ export class IndexingService {
); );
// console.log(document); // console.log(document);
await this.searchService.addDocuments('emails', [document], 'id'); await this.searchService.addDocuments('emails', [document], 'id');
} } */
/** /**
* Creates a search document from a raw email object and its attachments. * Creates a search document from a raw email object and its attachments.
*/ */
private async createEmailDocumentFromRawForBatch( /* private async createEmailDocumentFromRawForBatch(
email: EmailObject, email: EmailObject,
ingestionSourceId: string, ingestionSourceId: string,
archivedEmailId: string, archivedEmailId: string,
@@ -333,7 +328,7 @@ export class IndexingService {
timestamp: new Date(email.receivedAt).getTime(), timestamp: new Date(email.receivedAt).getTime(),
ingestionSourceId: ingestionSourceId, ingestionSourceId: ingestionSourceId,
}; };
} } */
private async createEmailDocumentFromRaw( private async createEmailDocumentFromRaw(
email: EmailObject, email: EmailObject,

View File

@@ -186,7 +186,7 @@ export class IngestionService {
(key) => (key) =>
key !== 'providerConfig' && key !== 'providerConfig' &&
originalSource[key as keyof IngestionSource] !== originalSource[key as keyof IngestionSource] !==
decryptedSource[key as keyof IngestionSource] decryptedSource[key as keyof IngestionSource]
); );
if (changedFields.length > 0) { if (changedFields.length > 0) {
await this.auditService.createAuditLog({ await this.auditService.createAuditLog({
@@ -518,12 +518,8 @@ export class IngestionService {
} }
} }
email.userEmail = userEmail;
return { return {
email, archivedEmailId: archivedEmail.id,
sourceId: source.id,
archivedId: archivedEmail.id,
}; };
} catch (error) { } catch (error) {
logger.error({ logger.error({

View File

@@ -81,6 +81,79 @@ export class StorageService implements IStorageProvider {
return Readable.from(decryptedContent); return Readable.from(decryptedContent);
} }
public async getStream(path: string): Promise<NodeJS.ReadableStream> {
const stream = await this.provider.get(path);
if (!this.encryptionKey) {
return stream;
}
// For encrypted files, we need to read the prefix and IV first.
// This part still buffers a small, fixed amount of data, which is acceptable.
const prefixAndIvBuffer = await new Promise<Buffer>((resolve, reject) => {
const chunks: Buffer[] = [];
let totalLength = 0;
const targetLength = ENCRYPTION_PREFIX.length + 16;
const onData = (chunk: Buffer) => {
chunks.push(chunk);
totalLength += chunk.length;
if (totalLength >= targetLength) {
stream.removeListener('data', onData);
resolve(Buffer.concat(chunks));
}
};
stream.on('data', onData);
stream.on('error', reject);
stream.on('end', () => {
// Handle cases where the file is smaller than the prefix + IV
if (totalLength < targetLength) {
resolve(Buffer.concat(chunks));
}
});
});
const prefix = prefixAndIvBuffer.subarray(0, ENCRYPTION_PREFIX.length);
if (!prefix.equals(ENCRYPTION_PREFIX)) {
// File is not encrypted, return a new stream containing the buffered prefix and the rest of the original stream
const combinedStream = new Readable({
read() { },
});
combinedStream.push(prefixAndIvBuffer);
stream.on('data', (chunk) => {
combinedStream.push(chunk);
});
stream.on('end', () => {
combinedStream.push(null); // No more data
});
stream.on('error', (err) => {
combinedStream.emit('error', err);
});
return combinedStream;
}
try {
const iv = prefixAndIvBuffer.subarray(
ENCRYPTION_PREFIX.length,
ENCRYPTION_PREFIX.length + 16
);
const decipher = createDecipheriv(this.algorithm, this.encryptionKey, iv);
// Push the remaining part of the initial buffer to the decipher
const remainingBuffer = prefixAndIvBuffer.subarray(ENCRYPTION_PREFIX.length + 16);
if (remainingBuffer.length > 0) {
decipher.write(remainingBuffer);
}
// Pipe the rest of the stream
stream.pipe(decipher);
return decipher;
} catch (error) {
throw new Error('Failed to decrypt file. It may be corrupted or the key is incorrect.');
}
}
delete(path: string): Promise<void> { delete(path: string): Promise<void> {
return this.provider.delete(path); return this.provider.delete(path);
} }

View File

@@ -131,6 +131,7 @@ export class ImapConnector implements IEmailConnector {
} catch (err: any) { } catch (err: any) {
logger.error({ err, attempt }, `IMAP operation failed on attempt ${attempt}`); logger.error({ err, attempt }, `IMAP operation failed on attempt ${attempt}`);
this.isConnected = false; // Force reconnect on next attempt this.isConnected = false; // Force reconnect on next attempt
this.client = this.createClient(); // Create a new client instance for the next retry
if (attempt === maxRetries) { if (attempt === maxRetries) {
logger.error({ err }, 'IMAP operation failed after all retries.'); logger.error({ err }, 'IMAP operation failed after all retries.');
throw err; throw err;

View File

@@ -10,9 +10,46 @@ import { simpleParser, ParsedMail, Attachment, AddressObject } from 'mailparser'
import { logger } from '../../config/logger'; import { logger } from '../../config/logger';
import { getThreadId } from './helpers/utils'; import { getThreadId } from './helpers/utils';
import { StorageService } from '../StorageService'; import { StorageService } from '../StorageService';
import { Readable } from 'stream'; import { Readable, Transform } from 'stream';
import { createHash } from 'crypto'; import { createHash } from 'crypto';
import { streamToBuffer } from '../../helpers/streamToBuffer';
class MboxSplitter extends Transform {
private buffer: Buffer = Buffer.alloc(0);
private delimiter: Buffer = Buffer.from('\nFrom ');
private firstChunk: boolean = true;
_transform(chunk: Buffer, encoding: string, callback: Function) {
if (this.firstChunk) {
// Check if the file starts with "From ". If not, prepend it to the first email.
if (chunk.subarray(0, 5).toString() !== 'From ') {
this.push(Buffer.from('From '));
}
this.firstChunk = false;
}
let currentBuffer = Buffer.concat([this.buffer, chunk]);
let position;
while ((position = currentBuffer.indexOf(this.delimiter)) > -1) {
const email = currentBuffer.subarray(0, position);
if (email.length > 0) {
this.push(email);
}
// The next email starts with "From ", which is what the parser expects.
currentBuffer = currentBuffer.subarray(position + 1);
}
this.buffer = currentBuffer;
callback();
}
_flush(callback: Function) {
if (this.buffer.length > 0) {
this.push(this.buffer);
}
callback();
}
}
export class MboxConnector implements IEmailConnector { export class MboxConnector implements IEmailConnector {
private storage: StorageService; private storage: StorageService;
@@ -57,30 +94,15 @@ export class MboxConnector implements IEmailConnector {
userEmail: string, userEmail: string,
syncState?: SyncState | null syncState?: SyncState | null
): AsyncGenerator<EmailObject | null> { ): AsyncGenerator<EmailObject | null> {
const fileStream = await this.storage.getStream(this.credentials.uploadedFilePath);
const mboxSplitter = new MboxSplitter();
const emailStream = fileStream.pipe(mboxSplitter);
try { try {
const fileStream = await this.storage.get(this.credentials.uploadedFilePath); for await (const emailBuffer of emailStream) {
const fileBuffer = await streamToBuffer(fileStream as Readable);
const mboxContent = fileBuffer.toString('utf-8');
const emailDelimiter = '\nFrom ';
const emails = mboxContent.split(emailDelimiter);
// The first split part might be empty or part of the first email's header, so we adjust.
if (emails.length > 0 && !mboxContent.startsWith('From ')) {
emails.shift(); // Adjust if the file doesn't start with "From "
}
logger.info(`Found ${emails.length} potential emails in the mbox file.`);
let emailCount = 0;
for (const email of emails) {
try { try {
// Re-add the "From " delimiter for the parser, except for the very first email const emailObject = await this.parseMessage(emailBuffer as Buffer, '');
const emailWithDelimiter =
emailCount > 0 || mboxContent.startsWith('From ') ? `From ${email}` : email;
const emailBuffer = Buffer.from(emailWithDelimiter, 'utf-8');
const emailObject = await this.parseMessage(emailBuffer, '');
yield emailObject; yield emailObject;
emailCount++;
} catch (error) { } catch (error) {
logger.error( logger.error(
{ error, file: this.credentials.uploadedFilePath }, { error, file: this.credentials.uploadedFilePath },
@@ -88,8 +110,31 @@ export class MboxConnector implements IEmailConnector {
); );
} }
} }
logger.info(`Finished processing mbox file. Total emails processed: ${emailCount}`);
} finally { } finally {
// Ensure all streams are properly closed before deleting the file.
if (fileStream instanceof Readable) {
fileStream.destroy();
}
if (emailStream instanceof Readable) {
emailStream.destroy();
}
// Wait for the streams to fully close to prevent race conditions with file deletion.
await new Promise((resolve) => {
if (fileStream instanceof Readable) {
fileStream.on('close', resolve);
} else {
resolve(true);
}
});
await new Promise((resolve) => {
if (emailStream instanceof Readable) {
emailStream.on('close', resolve);
} else {
resolve(true);
}
});
try { try {
await this.storage.delete(this.credentials.uploadedFilePath); await this.storage.delete(this.credentials.uploadedFilePath);
} catch (error) { } catch (error) {

View File

@@ -13,15 +13,8 @@ import { getThreadId } from './helpers/utils';
import { StorageService } from '../StorageService'; import { StorageService } from '../StorageService';
import { Readable } from 'stream'; import { Readable } from 'stream';
import { createHash } from 'crypto'; import { createHash } from 'crypto';
import { join } from 'path';
const streamToBuffer = (stream: Readable): Promise<Buffer> => { import { createWriteStream, promises as fs } from 'fs';
return new Promise((resolve, reject) => {
const chunks: Buffer[] = [];
stream.on('data', (chunk) => chunks.push(chunk));
stream.on('error', reject);
stream.on('end', () => resolve(Buffer.concat(chunks)));
});
};
// We have to hardcode names for deleted and trash folders here as current lib doesn't support looking into PST properties. // We have to hardcode names for deleted and trash folders here as current lib doesn't support looking into PST properties.
const DELETED_FOLDERS = new Set([ const DELETED_FOLDERS = new Set([
@@ -113,20 +106,25 @@ const JUNK_FOLDERS = new Set([
export class PSTConnector implements IEmailConnector { export class PSTConnector implements IEmailConnector {
private storage: StorageService; private storage: StorageService;
private pstFile: PSTFile | null = null;
constructor(private credentials: PSTImportCredentials) { constructor(private credentials: PSTImportCredentials) {
this.storage = new StorageService(); this.storage = new StorageService();
} }
private async loadPstFile(): Promise<PSTFile> { private async loadPstFile(): Promise<{ pstFile: PSTFile; tempDir: string }> {
if (this.pstFile) { const fileStream = await this.storage.getStream(this.credentials.uploadedFilePath);
return this.pstFile; const tempDir = await fs.mkdtemp(join('/tmp', `pst-import-${new Date().getTime()}`));
} const tempFilePath = join(tempDir, 'temp.pst');
const fileStream = await this.storage.get(this.credentials.uploadedFilePath);
const buffer = await streamToBuffer(fileStream as Readable); await new Promise<void>((resolve, reject) => {
this.pstFile = new PSTFile(buffer); const dest = createWriteStream(tempFilePath);
return this.pstFile; fileStream.pipe(dest);
dest.on('finish', resolve);
dest.on('error', reject);
});
const pstFile = new PSTFile(tempFilePath);
return { pstFile, tempDir };
} }
public async testConnection(): Promise<boolean> { public async testConnection(): Promise<boolean> {
@@ -156,8 +154,11 @@ export class PSTConnector implements IEmailConnector {
*/ */
public async *listAllUsers(): AsyncGenerator<MailboxUser> { public async *listAllUsers(): AsyncGenerator<MailboxUser> {
let pstFile: PSTFile | null = null; let pstFile: PSTFile | null = null;
let tempDir: string | null = null;
try { try {
pstFile = await this.loadPstFile(); const loadResult = await this.loadPstFile();
pstFile = loadResult.pstFile;
tempDir = loadResult.tempDir;
const root = pstFile.getRootFolder(); const root = pstFile.getRootFolder();
const displayName: string = const displayName: string =
root.displayName || pstFile.pstFilename || String(new Date().getTime()); root.displayName || pstFile.pstFilename || String(new Date().getTime());
@@ -171,10 +172,12 @@ export class PSTConnector implements IEmailConnector {
}; };
} catch (error) { } catch (error) {
logger.error({ error }, 'Failed to list users from PST file.'); logger.error({ error }, 'Failed to list users from PST file.');
pstFile?.close();
throw error; throw error;
} finally { } finally {
pstFile?.close(); pstFile?.close();
if (tempDir) {
await fs.rm(tempDir, { recursive: true, force: true });
}
} }
} }
@@ -183,16 +186,21 @@ export class PSTConnector implements IEmailConnector {
syncState?: SyncState | null syncState?: SyncState | null
): AsyncGenerator<EmailObject | null> { ): AsyncGenerator<EmailObject | null> {
let pstFile: PSTFile | null = null; let pstFile: PSTFile | null = null;
let tempDir: string | null = null;
try { try {
pstFile = await this.loadPstFile(); const loadResult = await this.loadPstFile();
pstFile = loadResult.pstFile;
tempDir = loadResult.tempDir;
const root = pstFile.getRootFolder(); const root = pstFile.getRootFolder();
yield* this.processFolder(root, '', userEmail); yield* this.processFolder(root, '', userEmail);
} catch (error) { } catch (error) {
logger.error({ error }, 'Failed to fetch email.'); logger.error({ error }, 'Failed to fetch email.');
pstFile?.close();
throw error; throw error;
} finally { } finally {
pstFile?.close(); pstFile?.close();
if (tempDir) {
await fs.rm(tempDir, { recursive: true, force: true });
}
try { try {
await this.storage.delete(this.credentials.uploadedFilePath); await this.storage.delete(this.credentials.uploadedFilePath);
} catch (error) { } catch (error) {
@@ -281,8 +289,8 @@ export class PSTConnector implements IEmailConnector {
emlBuffer ?? Buffer.from(parsedEmail.text || parsedEmail.html || '', 'utf-8') emlBuffer ?? Buffer.from(parsedEmail.text || parsedEmail.html || '', 'utf-8')
) )
.digest('hex')}-${createHash('sha256') .digest('hex')}-${createHash('sha256')
.update(emlBuffer ?? Buffer.from(msg.subject || '', 'utf-8')) .update(emlBuffer ?? Buffer.from(msg.subject || '', 'utf-8'))
.digest('hex')}-${msg.clientSubmitTime?.getTime()}`; .digest('hex')}-${msg.clientSubmitTime?.getTime()}`;
} }
return { return {
id: messageId, id: messageId,

View File

@@ -58,7 +58,7 @@
<Card.Root> <Card.Root>
<Card.Header> <Card.Header>
<Card.Title>{$t('app.jobs.jobs')}</Card.Title> <Card.Title>{$t('app.jobs.jobs')}</Card.Title>
<div class="flex space-x-2"> <div class="flex flex-wrap space-x-2 space-y-2">
{#each jobStatuses as status} {#each jobStatuses as status}
<Button <Button
variant={selectedStatus === status ? 'default' : 'outline'} variant={selectedStatus === status ? 'default' : 'outline'}

View File

@@ -56,13 +56,15 @@ export interface EmailObject {
} }
/** /**
* Represents an email that has been processed and is ready for indexing.
* Represents an email that has been processed and is ready for indexing. * Represents an email that has been processed and is ready for indexing.
* This interface defines the shape of the data that is passed to the batch indexing function. * This interface defines the shape of the data that is passed to the batch indexing function.
*/ */
export interface PendingEmail { export interface PendingEmail {
email: EmailObject; /** The unique identifier of the archived email record in the database.
sourceId: string; * This ID is used to retrieve the full email data from the database and storage for indexing.
archivedId: string; */
archivedEmailId: string;
} }
// Define the structure of the document to be indexed in Meilisearch // Define the structure of the document to be indexed in Meilisearch

69
pnpm-lock.yaml generated
View File

@@ -83,6 +83,12 @@ importers:
'@azure/msal-node': '@azure/msal-node':
specifier: ^3.6.3 specifier: ^3.6.3
version: 3.6.3 version: 3.6.3
'@bull-board/api':
specifier: ^6.14.0
version: 6.14.0(@bull-board/ui@6.14.0)
'@bull-board/express':
specifier: ^6.14.0
version: 6.14.0
'@casl/ability': '@casl/ability':
specifier: ^6.7.3 specifier: ^6.7.3
version: 6.7.3 version: 6.7.3
@@ -655,6 +661,17 @@ packages:
resolution: {integrity: sha512-ruv7Ae4J5dUYULmeXw1gmb7rYRz57OWCPM57pHojnLq/3Z1CK2lNSLTCVjxVk1F/TZHwOZZrOWi0ur95BbLxNQ==} resolution: {integrity: sha512-ruv7Ae4J5dUYULmeXw1gmb7rYRz57OWCPM57pHojnLq/3Z1CK2lNSLTCVjxVk1F/TZHwOZZrOWi0ur95BbLxNQ==}
engines: {node: '>=6.9.0'} engines: {node: '>=6.9.0'}
'@bull-board/api@6.14.0':
resolution: {integrity: sha512-oMDwXwoPn0RsdZ3Y68/bOErZ/qGZE5H97vgE/Pc8Uul/OHajlvajKW4NV+ZGTix82liUfH9CkjYx7PpwvBWhxg==}
peerDependencies:
'@bull-board/ui': 6.14.0
'@bull-board/express@6.14.0':
resolution: {integrity: sha512-3H1ame2G1+eVnqqSsw6KfzTGYAWSpVsIx6EPwg9vPSP2eKfNAm12Cm4zvL6ZkwAvTCkAByt5PPDRWbbwWB6HHQ==}
'@bull-board/ui@6.14.0':
resolution: {integrity: sha512-5yqfS9CwWR8DBxpReIbqv/VSPFM/zT4KZ75keyApMiejasRC2joaHqEzYWlMCjkMycbNNCvlQNlTbl+C3dE/dg==}
'@casl/ability@6.7.3': '@casl/ability@6.7.3':
resolution: {integrity: sha512-A4L28Ko+phJAsTDhRjzCOZWECQWN2jzZnJPnROWWHjJpyMq1h7h9ZqjwS2WbIUa3Z474X1ZPSgW0f1PboZGC0A==} resolution: {integrity: sha512-A4L28Ko+phJAsTDhRjzCOZWECQWN2jzZnJPnROWWHjJpyMq1h7h9ZqjwS2WbIUa3Z474X1ZPSgW0f1PboZGC0A==}
@@ -2724,6 +2741,11 @@ packages:
ee-first@1.1.1: ee-first@1.1.1:
resolution: {integrity: sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==} resolution: {integrity: sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==}
ejs@3.1.10:
resolution: {integrity: sha512-UeJmFfOrAQS8OJWPZ4qtgHyWExa088/MtK5UEyoJGFH67cDEXkZSviOiKRCZ4Xij0zxI3JECgYs3oKx+AizQBA==}
engines: {node: '>=0.10.0'}
hasBin: true
emoji-regex-xs@1.0.0: emoji-regex-xs@1.0.0:
resolution: {integrity: sha512-LRlerrMYoIDrT6jgpeZ2YYl/L8EulRTt5hQcYjy5AInh7HWXKimpqx68aknBFpGL2+/IcogTcaydJEgaTmOpDg==} resolution: {integrity: sha512-LRlerrMYoIDrT6jgpeZ2YYl/L8EulRTt5hQcYjy5AInh7HWXKimpqx68aknBFpGL2+/IcogTcaydJEgaTmOpDg==}
@@ -2879,6 +2901,9 @@ packages:
file-uri-to-path@1.0.0: file-uri-to-path@1.0.0:
resolution: {integrity: sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==} resolution: {integrity: sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==}
filelist@1.0.4:
resolution: {integrity: sha512-w1cEuf3S+DrLCQL7ET6kz+gmlJdbq9J7yXCSjK/OZCPA+qEN1WyF4ZAf0YYJa4/shHJra2t/d/r8SV4Ji+x+8Q==}
fill-range@7.1.1: fill-range@7.1.1:
resolution: {integrity: sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==} resolution: {integrity: sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==}
engines: {node: '>=8'} engines: {node: '>=8'}
@@ -3207,6 +3232,11 @@ packages:
jackspeak@3.4.3: jackspeak@3.4.3:
resolution: {integrity: sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw==} resolution: {integrity: sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw==}
jake@10.9.4:
resolution: {integrity: sha512-wpHYzhxiVQL+IV05BLE2Xn34zW1S223hvjtqk0+gsPrwd/8JNLXJgZZM/iPFsYc1xyphF+6M6EvdE5E9MBGkDA==}
engines: {node: '>=10'}
hasBin: true
jiti@2.4.2: jiti@2.4.2:
resolution: {integrity: sha512-rg9zJN+G4n2nfJl5MW3BMygZX56zKPNVEYYqq7adpmMh4Jn2QNEwhvQlFy6jPVdcod7txZtKHWnyZiA3a0zP7A==} resolution: {integrity: sha512-rg9zJN+G4n2nfJl5MW3BMygZX56zKPNVEYYqq7adpmMh4Jn2QNEwhvQlFy6jPVdcod7txZtKHWnyZiA3a0zP7A==}
hasBin: true hasBin: true
@@ -4010,6 +4040,9 @@ packages:
resolution: {integrity: sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==} resolution: {integrity: sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==}
engines: {node: '>=4'} engines: {node: '>=4'}
redis-info@3.1.0:
resolution: {integrity: sha512-ER4L9Sh/vm63DkIE0bkSjxluQlioBiBgf5w1UuldaW/3vPcecdljVDisZhmnCMvsxHNiARTTDDHGg9cGwTfrKg==}
redis-parser@3.0.0: redis-parser@3.0.0:
resolution: {integrity: sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==} resolution: {integrity: sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==}
engines: {node: '>=4'} engines: {node: '>=4'}
@@ -5361,6 +5394,24 @@ snapshots:
'@babel/helper-string-parser': 7.27.1 '@babel/helper-string-parser': 7.27.1
'@babel/helper-validator-identifier': 7.27.1 '@babel/helper-validator-identifier': 7.27.1
'@bull-board/api@6.14.0(@bull-board/ui@6.14.0)':
dependencies:
'@bull-board/ui': 6.14.0
redis-info: 3.1.0
'@bull-board/express@6.14.0':
dependencies:
'@bull-board/api': 6.14.0(@bull-board/ui@6.14.0)
'@bull-board/ui': 6.14.0
ejs: 3.1.10
express: 5.1.0
transitivePeerDependencies:
- supports-color
'@bull-board/ui@6.14.0':
dependencies:
'@bull-board/api': 6.14.0(@bull-board/ui@6.14.0)
'@casl/ability@6.7.3': '@casl/ability@6.7.3':
dependencies: dependencies:
'@ucast/mongo2js': 1.4.0 '@ucast/mongo2js': 1.4.0
@@ -7275,6 +7326,10 @@ snapshots:
ee-first@1.1.1: {} ee-first@1.1.1: {}
ejs@3.1.10:
dependencies:
jake: 10.9.4
emoji-regex-xs@1.0.0: {} emoji-regex-xs@1.0.0: {}
emoji-regex@8.0.0: {} emoji-regex@8.0.0: {}
@@ -7496,6 +7551,10 @@ snapshots:
file-uri-to-path@1.0.0: {} file-uri-to-path@1.0.0: {}
filelist@1.0.4:
dependencies:
minimatch: 5.1.6
fill-range@7.1.1: fill-range@7.1.1:
dependencies: dependencies:
to-regex-range: 5.0.1 to-regex-range: 5.0.1
@@ -7891,6 +7950,12 @@ snapshots:
optionalDependencies: optionalDependencies:
'@pkgjs/parseargs': 0.11.0 '@pkgjs/parseargs': 0.11.0
jake@10.9.4:
dependencies:
async: 3.2.6
filelist: 1.0.4
picocolors: 1.1.1
jiti@2.4.2: {} jiti@2.4.2: {}
jose@6.0.11: {} jose@6.0.11: {}
@@ -8688,6 +8753,10 @@ snapshots:
redis-errors@1.2.0: {} redis-errors@1.2.0: {}
redis-info@3.1.0:
dependencies:
lodash: 4.17.21
redis-parser@3.0.0: redis-parser@3.0.0:
dependencies: dependencies:
redis-errors: 1.2.0 redis-errors: 1.2.0