Ingestion database error fix, UI update

2026-04-06 00:31:57 +02:00 · 2025-08-01 15:09:05 +03:00
parent 488df16f26
commit 5cc24d0d67
12 changed files with 187 additions and 63 deletions
--- a/README.md
+++ b/README.md
@@ -19,6 +19,12 @@ _Archived emails_
 ![Open Archiver Preview](assets/screenshots/search.png)
 _Full-text search across all your emails and attachments_

+## Community
+
+Join our growing community on Discord to ask questions, share your projects, and connect with other developers.
+
+[![Discord](https://img.shields.io/badge/Join%20our%20Discord-7289DA?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/Qpv4BmHp)
+
 ## Key Features

 -   **Universal Ingestion**: Connect to Google Workspace, Microsoft 365, and standard IMAP servers to perform initial bulk imports and maintain continuous, real-time synchronization.
@@ -79,15 +85,9 @@ Open Archiver is built on a modern, scalable, and maintainable technology stack:

 After deploying the application, you will need to configure one or more ingestion sources to begin archiving emails. Follow our detailed guides to connect to your email provider:

-   [Connecting to Google Workspace](docs/user-guides/email-providers/google-workspace.md)
-   [Connecting to Microsoft 365](docs/user-guides/email-providers/microsoft-365.md)
-   [Connecting to a Generic IMAP Server](docs/user-guides/email-providers/imap.md)
-
-## Community
-
-Join our growing community on Discord to ask questions, share your projects, and connect with other developers.
-
-[![Discord](https://img.shields.io/badge/Join%20our%20Discord-7289DA?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/Qpv4BmHp)
+-   [Connecting to Google Workspace](https://docs.openarchiver.com/user-guides/email-providers/google-workspace.html)
+-   [Connecting to Microsoft 365](https://docs.openarchiver.com/user-guides/email-providers/imap.html)
+-   [Connecting to a Generic IMAP Server](https://docs.openarchiver.com/user-guides/email-providers/imap.html)

 ## Contributing

@@ -98,3 +98,5 @@ We welcome contributions from the community!
 -   **Code Contributions**: If you'd like to contribute code, please fork the repository and submit a pull request.

 Please read our `CONTRIBUTING.md` file for more details on our code of conduct and the process for submitting pull requests.
+
+## Star History [![Star History Chart](https://api.star-history.com/svg?repos=LogicLabs-OU/OpenArchiver&type=Date)](https://www.star-history.com/#LogicLabs-OU/OpenArchiver&Date)
--- a/docs/user-guides/installation.md
+++ b/docs/user-guides/installation.md
@@ -47,7 +47,7 @@ You must change the following placeholder values to secure your instance:

 By default, the Docker Compose setup uses local filesystem storage, which is persisted using a Docker volume named `archiver-data`. This is suitable for most use cases.

-If you want to use S3-compatible object storage, change the `STORAGE_TYPE` to `s3` and fill in your S3 credentials (`STORAGE_S3_*` variables).
+If you want to use S3-compatible object storage, change the `STORAGE_TYPE` to `s3` and fill in your S3 credentials (`STORAGE_S3_*` variables). When `STORAGE_TYPE` is set to `local`, the S3-related variables are not required.

 ### Using External Services

@@ -91,16 +91,16 @@ These variables are used by `docker-compose.yml` to configure the services.

 #### Storage Settings

-| Variable                       | Description                                      | Default Value             |
-| ------------------------------ | ------------------------------------------------ | ------------------------- |
-| `STORAGE_TYPE`                 | The storage backend to use (`local` or `s3`).    | `local`                   |
-| `STORAGE_LOCAL_ROOT_PATH`      | The root path for local file storage.            | `/var/data/open-archiver` |
-| `STORAGE_S3_ENDPOINT`          | The endpoint for S3-compatible storage.          |                           |
-| `STORAGE_S3_BUCKET`            | The bucket name for S3-compatible storage.       |                           |
-| `STORAGE_S3_ACCESS_KEY_ID`     | The access key ID for S3-compatible storage.     |                           |
-| `STORAGE_S3_SECRET_ACCESS_KEY` | The secret access key for S3-compatible storage. |                           |
-| `STORAGE_S3_REGION`            | The region for S3-compatible storage.            |                           |
-| `STORAGE_S3_FORCE_PATH_STYLE`  | Force path-style addressing for S3.              | `false`                   |
+| Variable                       | Description                                                                           | Default Value             |
+| ------------------------------ | ------------------------------------------------------------------------------------- | ------------------------- |
+| `STORAGE_TYPE`                 | The storage backend to use (`local` or `s3`).                                         | `local`                   |
+| `STORAGE_LOCAL_ROOT_PATH`      | The root path for local file storage.                                                 | `/var/data/open-archiver` |
+| `STORAGE_S3_ENDPOINT`          | The endpoint for S3-compatible storage (required if `STORAGE_TYPE` is `s3`).          |                           |
+| `STORAGE_S3_BUCKET`            | The bucket name for S3-compatible storage (required if `STORAGE_TYPE` is `s3`).       |                           |
+| `STORAGE_S3_ACCESS_KEY_ID`     | The access key ID for S3-compatible storage (required if `STORAGE_TYPE` is `s3`).     |                           |
+| `STORAGE_S3_SECRET_ACCESS_KEY` | The secret access key for S3-compatible storage (required if `STORAGE_TYPE` is `s3`). |                           |
+| `STORAGE_S3_REGION`            | The region for S3-compatible storage (required if `STORAGE_TYPE` is `s3`).            |                           |
+| `STORAGE_S3_FORCE_PATH_STYLE`  | Force path-style addressing for S3 (optional).                                        | `false`                   |

 #### Security & Authentication

--- a/packages/backend/src/services/IngestionService.ts
+++ b/packages/backend/src/services/IngestionService.ts
@@ -287,10 +287,13 @@ export class IngestionService {
                        })
                        .returning();

-                    await db.insert(emailAttachments).values({
-                        emailId: archivedEmail.id,
-                        attachmentId: newAttachment.id
-                    });
+                    await db
+                        .insert(emailAttachments)
+                        .values({
+                            emailId: archivedEmail.id,
+                            attachmentId: newAttachment.id
+                        })
+                        .onConflictDoNothing();
                }
            }
            // adding to indexing queue
--- a/packages/backend/src/services/ingestion-connectors/ImapConnector.ts
+++ b/packages/backend/src/services/ingestion-connectors/ImapConnector.ts
@@ -2,6 +2,7 @@ import type { GenericImapCredentials, EmailObject, EmailAddress, SyncState, Mail
 import type { IEmailConnector } from '../EmailProviderFactory';
 import { ImapFlow } from 'imapflow';
 import { simpleParser, ParsedMail, Attachment, AddressObject } from 'mailparser';
+import { logger } from '../../config/logger';

 export class ImapConnector implements IEmailConnector {
    private client: ImapFlow;
@@ -17,12 +18,12 @@ export class ImapConnector implements IEmailConnector {
                user: this.credentials.username,
                pass: this.credentials.password,
            },
-            logger: false, // Set to true for verbose logging
+            logger: logger.child({ module: 'ImapFlow' }),
        });

        // Handles client-level errors, like unexpected disconnects, to prevent crashes.
        this.client.on('error', (err) => {
-            console.error('IMAP client error:', err);
+            logger.error({ err }, 'IMAP client error');
            this.isConnected = false;
        });
    }
@@ -39,7 +40,7 @@ export class ImapConnector implements IEmailConnector {
            this.isConnected = true;
        } catch (err) {
            this.isConnected = false;
-            console.error('IMAP connection failed:', err);
+            logger.error({ err }, 'IMAP connection failed');
            throw err;
        }
    }
@@ -60,7 +61,7 @@ export class ImapConnector implements IEmailConnector {
            await this.disconnect();
            return true;
        } catch (error) {
-            console.error('Failed to verify IMAP connection:', error);
+            logger.error({ error }, 'Failed to verify IMAP connection');
            return false;
        }
    }
@@ -101,10 +102,10 @@ export class ImapConnector implements IEmailConnector {
                await this.connect();
                return await action();
            } catch (err: any) {
-                console.error(`IMAP operation failed on attempt ${attempt}:`, err.message);
+                logger.error({ err, attempt }, `IMAP operation failed on attempt ${attempt}`);
                this.isConnected = false; // Force reconnect on next attempt
                if (attempt === maxRetries) {
-                    console.error('IMAP operation failed after all retries.');
+                    logger.error({ err }, 'IMAP operation failed after all retries.');
                    throw err;
                }
                // Wait for a short period before retrying
--- a/packages/frontend/src/app.css
+++ b/packages/frontend/src/app.css
@@ -24,11 +24,11 @@
 	--border: oklch(0.92 0.004 286.32);
 	--input: oklch(0.92 0.004 286.32);
 	--ring: oklch(0.705 0.213 47.604);
-	--chart-1: oklch(0.646 0.222 41.116);
-	--chart-2: oklch(0.705 0.213 47.604);
-	--chart-3: oklch(0.837 0.128 66.29);
-	--chart-4: oklch(0.553 0.195 38.402);
-	--chart-5: oklch(0.47 0.157 37.304);
+	--chart-1: oklch(0.705 0.213 47.604);
+	--chart-2: oklch(0.5093 0.0758 213.43);
+	--chart-3: oklch(0.9227 0.0517 91.38);
+	--chart-4: oklch(0.7509 0.1563 46.19);
+	--chart-5: oklch(0.7145 0.1478 266.89);
 	--sidebar: oklch(0.985 0 0);
 	--sidebar-foreground: oklch(0.141 0.005 285.823);
 	--sidebar-primary: oklch(0.705 0.213 47.604);
@@ -58,11 +58,11 @@
 	--border: oklch(1 0 0 / 10%);
 	--input: oklch(1 0 0 / 15%);
 	--ring: oklch(0.646 0.222 41.116);
-	--chart-1: oklch(0.646 0.222 41.116);
-	--chart-2: oklch(0.705 0.213 47.604);
-	--chart-3: oklch(0.837 0.128 66.29);
-	--chart-4: oklch(0.553 0.195 38.402);
-	--chart-5: oklch(0.47 0.157 37.304);
+	--chart-1: oklch(0.705 0.213 47.604);
+	--chart-2: oklch(0.5093 0.0758 213.43);
+	--chart-3: oklch(0.9227 0.0517 91.38);
+	--chart-4: oklch(0.7509 0.1563 46.19);
+	--chart-5: oklch(0.7145 0.1478 266.89);
 	--sidebar: oklch(0.21 0.006 285.885);
 	--sidebar-foreground: oklch(0.985 0 0);
 	--sidebar-primary: oklch(0.646 0.222 41.116);
--- a/packages/frontend/src/lib/components/custom/EmptyState.svelte
+++ b/packages/frontend/src/lib/components/custom/EmptyState.svelte
@@ -1,7 +1,5 @@
 <script lang="ts">
-	import { Button, buttonVariants } from '$lib/components/ui/button/index.js';
-	import * as Dialog from '$lib/components/ui/dialog/index.js';
-	import { type Snippet } from 'svelte';
+	import { Button } from '$lib/components/ui/button/index.js';

 	let {
 		header,
@@ -35,8 +33,8 @@
 		</svg>
 	</div>

-	<h3 class="mt-2 text-sm font-semibold text-gray-900">{header}</h3>
-	<p class="mt-1 text-sm text-gray-500">{text}</p>
+	<h3 class="0 mt-2 text-sm font-semibold">{header}</h3>
+	<p class="mt-1 text-sm">{text}</p>
 	<div>
 		<Button
 			variant="outline"
--- a/packages/frontend/src/lib/components/custom/IngestionSourceForm.svelte
+++ b/packages/frontend/src/lib/components/custom/IngestionSourceForm.svelte
@@ -6,6 +6,7 @@
 	import { Input } from '$lib/components/ui/input';
 	import { Label } from '$lib/components/ui/label';
 	import * as Select from '$lib/components/ui/select';
+	import * as Alert from '$lib/components/ui/alert/index.js';
 	import { Textarea } from '$lib/components/ui/textarea/index.js';

 	let {
@@ -17,16 +18,16 @@
 	} = $props();

 	const providerOptions = [
+		{ value: 'generic_imap', label: 'Generic IMAP' },
 		{ value: 'google_workspace', label: 'Google Workspace' },
-		{ value: 'microsoft_365', label: 'Microsoft 365' },
-		{ value: 'generic_imap', label: 'Generic IMAP' }
+		{ value: 'microsoft_365', label: 'Microsoft 365' }
 	];

 	let formData: CreateIngestionSourceDto = $state({
 		name: source?.name ?? '',
-		provider: source?.provider ?? 'google_workspace',
+		provider: source?.provider ?? 'generic_imap',
 		providerConfig: source?.credentials ?? {
-			type: source?.provider ?? 'google_workspace',
+			type: source?.provider ?? 'generic_imap',
 			secure: true
 		}
 	});
@@ -55,11 +56,11 @@

 <form onsubmit={handleSubmit} class="grid gap-4 py-4">
 	<div class="grid grid-cols-4 items-center gap-4">
-		<Label for="name" class="text-right">Name</Label>
+		<Label for="name" class="text-left">Name</Label>
 		<Input id="name" bind:value={formData.name} class="col-span-3" />
 	</div>
 	<div class="grid grid-cols-4 items-center gap-4">
-		<Label for="provider" class="text-right">Provider</Label>
+		<Label for="provider" class="text-left">Provider</Label>
 		<Select.Root name="provider" bind:value={formData.provider} type="single">
 			<Select.Trigger class="col-span-3">
 				{triggerContent}
@@ -74,7 +75,7 @@

 	{#if formData.provider === 'google_workspace'}
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="serviceAccountKeyJson" class="text-right">Service Account Key (JSON)</Label>
+			<Label for="serviceAccountKeyJson" class="text-left">Service Account Key (JSON)</Label>
 			<Textarea
 				placeholder="Paste your service account key JSON content"
 				id="serviceAccountKeyJson"
@@ -83,7 +84,7 @@
 			/>
 		</div>
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="impersonatedAdminEmail" class="text-right">Impersonated Admin Email</Label>
+			<Label for="impersonatedAdminEmail" class="text-left">Impersonated Admin Email</Label>
 			<Input
 				id="impersonatedAdminEmail"
 				bind:value={formData.providerConfig.impersonatedAdminEmail}
@@ -92,11 +93,11 @@
 		</div>
 	{:else if formData.provider === 'microsoft_365'}
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="clientId" class="text-right">Application (Client) ID</Label>
+			<Label for="clientId" class="text-left">Application (Client) ID</Label>
 			<Input id="clientId" bind:value={formData.providerConfig.clientId} class="col-span-3" />
 		</div>
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="clientSecret" class="text-right">Client Secret Value</Label>
+			<Label for="clientSecret" class="text-left">Client Secret Value</Label>
 			<Input
 				id="clientSecret"
 				type="password"
@@ -106,24 +107,24 @@
 			/>
 		</div>
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="tenantId" class="text-right">Directory (Tenant) ID</Label>
+			<Label for="tenantId" class="text-left">Directory (Tenant) ID</Label>
 			<Input id="tenantId" bind:value={formData.providerConfig.tenantId} class="col-span-3" />
 		</div>
 	{:else if formData.provider === 'generic_imap'}
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="host" class="text-right">Host</Label>
+			<Label for="host" class="text-left">Host</Label>
 			<Input id="host" bind:value={formData.providerConfig.host} class="col-span-3" />
 		</div>
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="port" class="text-right">Port</Label>
+			<Label for="port" class="text-left">Port</Label>
 			<Input id="port" type="number" bind:value={formData.providerConfig.port} class="col-span-3" />
 		</div>
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="username" class="text-right">Username</Label>
+			<Label for="username" class="text-left">Username</Label>
 			<Input id="username" bind:value={formData.providerConfig.username} class="col-span-3" />
 		</div>
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="password" class="text-right">Password</Label>
+			<Label for="password" class="text-left">Password</Label>
 			<Input
 				id="password"
 				type="password"
@@ -132,10 +133,22 @@
 			/>
 		</div>
 		<div class="grid grid-cols-4 items-center gap-4">
-			<Label for="secure" class="text-right">Use TLS</Label>
+			<Label for="secure" class="text-left">Use TLS</Label>
 			<Checkbox id="secure" bind:checked={formData.providerConfig.secure} />
 		</div>
 	{/if}
+	{#if formData.provider === 'google_workspace' || formData.provider === 'microsoft_365'}
+		<Alert.Root>
+			<Alert.Title>Heads up!</Alert.Title>
+			<Alert.Description>
+				<div class="my-1">
+					Please note that this is an organization-wide operation. This kind of ingestions will
+					import and index <b>all</b> email inboxes in your organization. If you want to import only
+					specific email inboxes, use the IMAP connector.
+				</div>
+			</Alert.Description>
+		</Alert.Root>
+	{/if}
 	<Dialog.Footer>
 		<Button type="submit" disabled={isSubmitting}>
 			{#if isSubmitting}
--- a/packages/frontend/src/lib/components/ui/alert/alert-description.svelte
+++ b/packages/frontend/src/lib/components/ui/alert/alert-description.svelte
@@ -0,0 +1,23 @@
+<script lang="ts">
+	import type { HTMLAttributes } from "svelte/elements";
+	import { cn, type WithElementRef } from "$lib/utils.js";
+
+	let {
+		ref = $bindable(null),
+		class: className,
+		children,
+		...restProps
+	}: WithElementRef<HTMLAttributes<HTMLDivElement>> = $props();
+</script>
+
+<div
+	bind:this={ref}
+	data-slot="alert-description"
+	class={cn(
+		"text-muted-foreground col-start-2 grid justify-items-start gap-1 text-sm [&_p]:leading-relaxed",
+		className
+	)}
+	{...restProps}
+>
+	{@render children?.()}
+</div>
--- a/packages/frontend/src/lib/components/ui/alert/alert-title.svelte
+++ b/packages/frontend/src/lib/components/ui/alert/alert-title.svelte
@@ -0,0 +1,20 @@
+<script lang="ts">
+	import type { HTMLAttributes } from "svelte/elements";
+	import { cn, type WithElementRef } from "$lib/utils.js";
+
+	let {
+		ref = $bindable(null),
+		class: className,
+		children,
+		...restProps
+	}: WithElementRef<HTMLAttributes<HTMLDivElement>> = $props();
+</script>
+
+<div
+	bind:this={ref}
+	data-slot="alert-title"
+	class={cn("col-start-2 line-clamp-1 min-h-4 font-medium tracking-tight", className)}
+	{...restProps}
+>
+	{@render children?.()}
+</div>
--- a/packages/frontend/src/lib/components/ui/alert/alert.svelte
+++ b/packages/frontend/src/lib/components/ui/alert/alert.svelte
@@ -0,0 +1,44 @@
+<script lang="ts" module>
+	import { type VariantProps, tv } from "tailwind-variants";
+
+	export const alertVariants = tv({
+		base: "relative grid w-full grid-cols-[0_1fr] items-start gap-y-0.5 rounded-lg border px-4 py-3 text-sm has-[>svg]:grid-cols-[calc(var(--spacing)*4)_1fr] has-[>svg]:gap-x-3 [&>svg]:size-4 [&>svg]:translate-y-0.5 [&>svg]:text-current",
+		variants: {
+			variant: {
+				default: "bg-card text-card-foreground",
+				destructive:
+					"text-destructive bg-card *:data-[slot=alert-description]:text-destructive/90 [&>svg]:text-current",
+			},
+		},
+		defaultVariants: {
+			variant: "default",
+		},
+	});
+
+	export type AlertVariant = VariantProps<typeof alertVariants>["variant"];
+</script>
+
+<script lang="ts">
+	import type { HTMLAttributes } from "svelte/elements";
+	import { cn, type WithElementRef } from "$lib/utils.js";
+
+	let {
+		ref = $bindable(null),
+		class: className,
+		variant = "default",
+		children,
+		...restProps
+	}: WithElementRef<HTMLAttributes<HTMLDivElement>> & {
+		variant?: AlertVariant;
+	} = $props();
+</script>
+
+<div
+	bind:this={ref}
+	data-slot="alert"
+	class={cn(alertVariants({ variant }), className)}
+	{...restProps}
+	role="alert"
+>
+	{@render children?.()}
+</div>
--- a/packages/frontend/src/lib/components/ui/alert/index.ts
+++ b/packages/frontend/src/lib/components/ui/alert/index.ts
@@ -0,0 +1,14 @@
+import Root from "./alert.svelte";
+import Description from "./alert-description.svelte";
+import Title from "./alert-title.svelte";
+export { alertVariants, type AlertVariant } from "./alert.svelte";
+
+export {
+	Root,
+	Description,
+	Title,
+	//
+	Root as Alert,
+	Description as AlertDescription,
+	Title as AlertTitle,
+};
--- a/packages/frontend/src/routes/dashboard/ingestions/+page.svelte
+++ b/packages/frontend/src/routes/dashboard/ingestions/+page.svelte
@@ -10,7 +10,6 @@
 	import { api } from '$lib/api.client';
 	import type { IngestionSource, CreateIngestionSourceDto } from '@open-archiver/types';
 	import Badge from '$lib/components/ui/badge/badge.svelte';
-	import type { BadgeVariant } from '$lib/components/ui/badge/badge.svelte';

 	let { data }: { data: PageData } = $props();

@@ -202,13 +201,20 @@
 </div>

 <Dialog.Root bind:open={isDialogOpen}>
-	<Dialog.Content class="sm:max-w-[425px]">
+	<Dialog.Content class="sm:max-w-120 md:max-w-180">
 		<Dialog.Header>
 			<Dialog.Title>{selectedSource ? 'Edit' : 'Create'} Ingestion Source</Dialog.Title>
 			<Dialog.Description>
 				{selectedSource
 					? 'Make changes to your ingestion source here.'
 					: 'Add a new ingestion source to start archiving emails.'}
+				<span
+					>Read <a
+						class="text-primary underline underline-offset-2"
+						target="_blank"
+						href="https://docs.openarchiver.com/user-guides/email-providers/">docs here</a
+					>.</span
+				>
 			</Dialog.Description>
 		</Dialog.Header>
 		<IngestionSourceForm source={selectedSource} onSubmit={handleFormSubmit} />