Database migration. Adding ingestion service

This commit is contained in:
Wayne
2025-07-11 13:36:34 +03:00
parent cc08f35ada
commit 3eb155ee16
74 changed files with 2947 additions and 21 deletions

View File

@@ -4,9 +4,6 @@ PORT_BACKEND=4000
PORT_FRONTEND=3000
# PostgreSQL
POSTGRES_DB=open_archive
POSTGRES_USER=admin
POSTGRES_PASSWORD=password
DATABASE_URL="postgresql://admin:password@postgres:5432/open_archive?schema=public"
# Redis
@@ -25,4 +22,5 @@ JWT_EXPIRES_IN="7d"
# Admin users
ADMIN_EMAIL=admin@local.com
ADMIN_PASSWORD=a_strong_pass
ADMIN_PASSWORD=a_strong_pass
SUPER_API_KEY=

3
.gitignore vendored
View File

@@ -17,4 +17,5 @@ pnpm-debug.log
.DS_Store
# Dev
.dev
.dev
.clinerules

205
docs/api/ingestion.md Normal file
View File

@@ -0,0 +1,205 @@
# Ingestion Sources API Documentation
A comprehensive guide to using the Ingestion Sources API.
**Base Path:** `/v1/ingestion-sources`
---
## Authentication
All endpoints in this API are protected and require authentication. Requests must include an `Authorization` header containing a valid Bearer token. This can be a JWT obtained from the login endpoint or a `SUPER_API_KEY` for administrative tasks.
**Header Example:**
`Authorization: Bearer <YOUR_JWT_OR_SUPER_API_KEY>`
---
## Core Concepts
### Ingestion Providers
The `provider` field determines the type of email source. Each provider requires a different configuration object, for example:
- `google_workspace`: For connecting to Google Workspace accounts via OAuth 2.0.
- `microsoft_365`: For connecting to Microsoft 365 accounts via OAuth 2.0.
- `generic_imap`: For connecting to any email server that supports IMAP.
### Ingestion Status
The `status` field tracks the state of the ingestion source.
- `pending_auth`: The source has been created but requires user authorization (OAuth flow).
- `active`: The source is authenticated and ready to sync.
- `syncing`: An import job is currently in progress.
- `paused`: The source is temporarily disabled.
- `error`: An error occurred during the last sync.
---
## 1. Create Ingestion Source
- **Method:** `POST`
- **Path:** `/`
- **Description:** Registers a new source for email ingestion. The `providerConfig` will vary based on the selected `provider`.
#### Request Body (`CreateIngestionSourceDto`)
- `name` (string, required): A user-friendly name for the source (e.g., "Marketing Department G-Suite").
- `provider` (string, required): One of `google_workspace`, `microsoft_365`, or `generic_imap`.
- `providerConfig` (object, required): Configuration specific to the provider.
##### `providerConfig` for `google_workspace` / `microsoft_365`
```json
{
"name": "Corporate Google Workspace",
"provider": "google_workspace",
"providerConfig": {
"clientId": "your-oauth-client-id.apps.googleusercontent.com",
"clientSecret": "your-super-secret-client-secret",
"redirectUri": "https://yourapp.com/oauth/google/callback"
}
}
```
##### `providerConfig` for `generic_imap`
```json
{
"name": "Legacy IMAP Server",
"provider": "generic_imap",
"providerConfig": {
"host": "imap.example.com",
"port": 993,
"secure": true,
"username": "archive-user",
"password": "imap-password"
}
}
```
#### Responses
- **Success (`201 Created`):** Returns the full `IngestionSource` object, which now includes a system-generated `id` and default status.
```json
{
"id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
"name": "Corporate Google Workspace",
"provider": "google_workspace",
"status": "pending_auth",
"createdAt": "2025-07-11T12:00:00.000Z",
"updatedAt": "2025-07-11T12:00:00.000Z",
"providerConfig": { ... }
}
```
- **Error (`500 Internal Server Error`):** Indicates a server-side problem during creation.
---
## 2. Get All Ingestion Sources
- **Method:** `GET`
- **Path:** `/`
- **Description:** Retrieves a list of all configured ingestion sources for the organization.
#### Responses
- **Success (`200 OK`):** Returns an array of `IngestionSource` objects.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.
---
## 3. Get Ingestion Source by ID
- **Method:** `GET`
- **Path:** `/:id`
- **Description:** Fetches the details of a specific ingestion source.
#### URL Parameters
- `id` (string, required): The UUID of the ingestion source.
#### Responses
- **Success (`200 OK`):** Returns the corresponding `IngestionSource` object.
- **Error (`404 Not Found`):** Returned if no source with the given ID exists.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.
---
## 4. Update Ingestion Source
- **Method:** `PUT`
- **Path:** `/:id`
- **Description:** Modifies an existing ingestion source. This is useful for changing the name, pausing a source, or updating its configuration.
#### URL Parameters
- `id` (string, required): The UUID of the ingestion source to update.
#### Request Body (`UpdateIngestionSourceDto`)
All fields are optional. Only include the fields you want to change.
```json
{
"name": "Marketing Dept G-Suite (Paused)",
"status": "paused"
}
```
#### Responses
- **Success (`200 OK`):** Returns the complete, updated `IngestionSource` object.
- **Error (`404 Not Found`):** Returned if no source with the given ID exists.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.
---
## 5. Delete Ingestion Source
- **Method:** `DELETE`
- **Path:** `/:id`
- **Description:** Permanently removes an ingestion source. This action cannot be undone.
#### URL Parameters
- `id` (string, required): The UUID of the ingestion source to delete.
#### Responses
- **Success (`204 No Content`):** Indicates successful deletion with no body content.
- **Error (`404 Not Found`):** Returned if no source with the given ID exists.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.
---
## 6. Trigger Initial Import
- **Method:** `POST`
- **Path:** `/:id/sync`
- **Description:** Initiates the email import process for a given source. This is an asynchronous operation that enqueues a background job and immediately returns a response. The status of the source will be updated to `syncing`.
#### URL Parameters
- `id` (string, required): The UUID of the ingestion source to sync.
#### Responses
- **Success (`202 Accepted`):** Confirms that the sync request has been accepted for processing.
```json
{
"message": "Initial import triggered successfully."
}
```
- **Error (`404 Not Found`):** Returned if no source with the given ID exists.
- **Error (`500 Internal Server Error`):** Indicates a server-side problem.

View File

@@ -0,0 +1,19 @@
import { defineConfig } from 'drizzle-kit';
import { config } from 'dotenv';
config({ path: '../../.env' });
if (!process.env.DATABASE_URL) {
throw new Error('DATABASE_URL is not set in the .env file');
}
export default defineConfig({
schema: './src/database/schema.ts',
out: './src/database/migrations',
dialect: 'postgresql',
dbCredentials: {
url: process.env.DATABASE_URL,
},
verbose: true,
strict: true,
});

View File

@@ -7,16 +7,22 @@
"dev": "ts-node-dev --respawn --transpile-only src/index.ts ",
"build": "tsc",
"start": "node dist/index.js",
"start:worker": "ts-node-dev --respawn --transpile-only src/workers/ingestion.worker.ts"
"start:worker": "ts-node-dev --respawn --transpile-only src/workers/ingestion.worker.ts",
"db:generate": "drizzle-kit generate --config=drizzle.config.ts",
"db:push": "drizzle-kit push --config=drizzle.config.ts",
"db:migrate": "ts-node-dev src/database/migrate.ts"
},
"dependencies": {
"@open-archive/types": "workspace:*",
"bcryptjs": "^3.0.2",
"bullmq": "^5.56.3",
"dotenv": "^17.2.0",
"drizzle-orm": "^0.44.2",
"express": "^5.1.0",
"express-validator": "^7.2.1",
"jose": "^6.0.11",
"pg": "^8.16.3",
"postgres": "^3.4.7",
"reflect-metadata": "^0.2.2",
"sqlite3": "^5.1.7",
"tsconfig-paths": "^4.2.0"
@@ -24,6 +30,7 @@
"devDependencies": {
"@types/express": "^5.0.3",
"@types/node": "^24.0.12",
"drizzle-kit": "^0.31.4",
"ts-node-dev": "^2.0.0",
"typescript": "^5.8.3"
}

View File

@@ -0,0 +1,83 @@
import { Request, Response } from 'express';
import { IngestionService } from '../../services/IngestionService';
import { CreateIngestionSourceDto, UpdateIngestionSourceDto } from '@open-archive/types';
export class IngestionController {
public create = async (req: Request, res: Response): Promise<Response> => {
try {
const dto: CreateIngestionSourceDto = req.body;
const newSource = await IngestionService.create(dto);
return res.status(201).json(newSource);
} catch (error) {
console.error('Create ingestion source error:', error);
return res.status(500).json({ message: 'An internal server error occurred' });
}
};
public findAll = async (req: Request, res: Response): Promise<Response> => {
try {
const sources = await IngestionService.findAll();
return res.status(200).json(sources);
} catch (error) {
console.error('Find all ingestion sources error:', error);
return res.status(500).json({ message: 'An internal server error occurred' });
}
};
public findById = async (req: Request, res: Response): Promise<Response> => {
try {
const { id } = req.params;
const source = await IngestionService.findById(id);
return res.status(200).json(source);
} catch (error) {
console.error(`Find ingestion source by id ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message });
}
return res.status(500).json({ message: 'An internal server error occurred' });
}
};
public update = async (req: Request, res: Response): Promise<Response> => {
try {
const { id } = req.params;
const dto: UpdateIngestionSourceDto = req.body;
const updatedSource = await IngestionService.update(id, dto);
return res.status(200).json(updatedSource);
} catch (error) {
console.error(`Update ingestion source ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message });
}
return res.status(500).json({ message: 'An internal server error occurred' });
}
};
public delete = async (req: Request, res: Response): Promise<Response> => {
try {
const { id } = req.params;
await IngestionService.delete(id);
return res.status(204).send();
} catch (error) {
console.error(`Delete ingestion source ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message });
}
return res.status(500).json({ message: 'An internal server error occurred' });
}
};
public triggerInitialImport = async (req: Request, res: Response): Promise<Response> => {
try {
const { id } = req.params;
await IngestionService.triggerInitialImport(id);
return res.status(202).json({ message: 'Initial import triggered successfully.' });
} catch (error) {
console.error(`Trigger initial import for ${req.params.id} error:`, error);
if (error instanceof Error && error.message === 'Ingestion source not found') {
return res.status(404).json({ message: error.message });
}
return res.status(500).json({ message: 'An internal server error occurred' });
}
};
}

View File

@@ -1,7 +1,7 @@
import type { Request, Response, NextFunction } from 'express';
import type { IAuthService } from '../../services/AuthService';
import type { AuthTokenPayload } from '@open-archive/types';
import 'dotenv/config';
// By using module augmentation, we can add our custom 'user' property
// to the Express Request interface in a type-safe way.
declare global {
@@ -15,20 +15,20 @@ declare global {
export const requireAuth = (authService: IAuthService) => {
return async (req: Request, res: Response, next: NextFunction) => {
const authHeader = req.headers.authorization;
if (!authHeader || !authHeader.startsWith('Bearer ')) {
return res.status(401).json({ message: 'Unauthorized: No token provided' });
}
const token = authHeader.split(' ')[1];
try {
// use a SUPER_API_KEY for all authentications.
if (token === process.env.SUPER_API_KEY) {
next();
return;
}
const payload = await authService.verifyToken(token);
if (!payload) {
return res.status(401).json({ message: 'Unauthorized: Invalid token' });
}
req.user = payload;
next();
} catch (error) {

View File

@@ -0,0 +1,28 @@
import { Router } from 'express';
import { IngestionController } from '../controllers/ingestion.controller';
import { requireAuth } from '../middleware/requireAuth';
import { IAuthService } from '../../services/AuthService';
export const createIngestionRouter = (
ingestionController: IngestionController,
authService: IAuthService
): Router => {
const router = Router();
// Secure all routes in this module
router.use(requireAuth(authService));
router.post('/', ingestionController.create);
router.get('/', ingestionController.findAll);
router.get('/:id', ingestionController.findById);
router.put('/:id', ingestionController.update);
router.delete('/:id', ingestionController.delete);
router.post('/:id/sync', ingestionController.triggerInitialImport);
return router;
};

View File

@@ -0,0 +1,12 @@
import { drizzle } from 'drizzle-orm/postgres-js';
import postgres from 'postgres';
import 'dotenv/config';
import * as schema from './schema';
if (!process.env.DATABASE_URL) {
throw new Error('DATABASE_URL is not set in the .env file');
}
const client = postgres(process.env.DATABASE_URL);
export const db = drizzle(client, { schema });

View File

@@ -0,0 +1,27 @@
import { migrate } from 'drizzle-orm/postgres-js/migrator';
import { drizzle } from 'drizzle-orm/postgres-js';
import postgres from 'postgres';
import { config } from 'dotenv';
config({ path: '../../.env' });
const runMigrate = async () => {
if (!process.env.DATABASE_URL) {
throw new Error('DATABASE_URL is not set in the .env file');
}
const connection = postgres(process.env.DATABASE_URL, { max: 1 });
const db = drizzle(connection);
console.log('Running migrations...');
await migrate(db, { migrationsFolder: 'src/database/migrations' });
console.log('Migrations completed!');
process.exit(0);
};
runMigrate().catch((err) => {
console.error('Migration failed!', err);
process.exit(1);
});

View File

@@ -0,0 +1,125 @@
CREATE TYPE "public"."retention_action" AS ENUM('delete_permanently', 'notify_admin');--> statement-breakpoint
CREATE TYPE "public"."ingestion_provider" AS ENUM('google_workspace', 'microsoft_365', 'generic_imap');--> statement-breakpoint
CREATE TYPE "public"."ingestion_status" AS ENUM('active', 'paused', 'error', 'pending_auth', 'syncing');--> statement-breakpoint
CREATE TABLE "archived_emails" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"custodian_id" uuid NOT NULL,
"message_id_header" text NOT NULL,
"sent_at" timestamp with time zone NOT NULL,
"subject" text,
"sender_name" text,
"sender_email" text NOT NULL,
"recipients" jsonb,
"storage_path" text NOT NULL,
"storage_hash_sha256" text NOT NULL,
"size_bytes" bigint NOT NULL,
"is_indexed" boolean DEFAULT false NOT NULL,
"has_attachments" boolean DEFAULT false NOT NULL,
"is_on_legal_hold" boolean DEFAULT false NOT NULL,
"archived_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE "attachments" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"filename" text NOT NULL,
"mime_type" text,
"size_bytes" bigint NOT NULL,
"content_hash_sha256" text NOT NULL,
"storage_path" text NOT NULL,
CONSTRAINT "attachments_content_hash_sha256_unique" UNIQUE("content_hash_sha256")
);
--> statement-breakpoint
CREATE TABLE "email_attachments" (
"email_id" uuid NOT NULL,
"attachment_id" uuid NOT NULL,
CONSTRAINT "email_attachments_email_id_attachment_id_pk" PRIMARY KEY("email_id","attachment_id")
);
--> statement-breakpoint
CREATE TABLE "audit_logs" (
"id" bigserial PRIMARY KEY NOT NULL,
"timestamp" timestamp with time zone DEFAULT now() NOT NULL,
"actor_identifier" text NOT NULL,
"action" text NOT NULL,
"target_type" text,
"target_id" text,
"details" jsonb,
"is_tamper_evident" boolean DEFAULT false
);
--> statement-breakpoint
CREATE TABLE "ediscovery_cases" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"name" text NOT NULL,
"description" text,
"status" text DEFAULT 'open' NOT NULL,
"created_by_identifier" text NOT NULL,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL,
CONSTRAINT "ediscovery_cases_name_unique" UNIQUE("name")
);
--> statement-breakpoint
CREATE TABLE "export_jobs" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"case_id" uuid,
"format" text NOT NULL,
"status" text DEFAULT 'pending' NOT NULL,
"query" jsonb NOT NULL,
"file_path" text,
"created_by_identifier" text NOT NULL,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"completed_at" timestamp with time zone
);
--> statement-breakpoint
CREATE TABLE "legal_holds" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"case_id" uuid NOT NULL,
"custodian_id" uuid,
"hold_criteria" jsonb,
"reason" text,
"applied_by_identifier" text NOT NULL,
"applied_at" timestamp with time zone DEFAULT now() NOT NULL,
"removed_at" timestamp with time zone
);
--> statement-breakpoint
CREATE TABLE "retention_policies" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"name" text NOT NULL,
"description" text,
"priority" integer NOT NULL,
"retention_period_days" integer NOT NULL,
"action_on_expiry" "retention_action" NOT NULL,
"is_enabled" boolean DEFAULT true NOT NULL,
"conditions" jsonb,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL,
CONSTRAINT "retention_policies_name_unique" UNIQUE("name")
);
--> statement-breakpoint
CREATE TABLE "custodians" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"email" text NOT NULL,
"display_name" text,
"source_type" "ingestion_provider" NOT NULL,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL,
CONSTRAINT "custodians_email_unique" UNIQUE("email")
);
--> statement-breakpoint
CREATE TABLE "ingestion_sources" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"name" text NOT NULL,
"provider" "ingestion_provider" NOT NULL,
"credentials" jsonb,
"status" "ingestion_status" DEFAULT 'pending_auth' NOT NULL,
"last_sync_started_at" timestamp with time zone,
"last_sync_finished_at" timestamp with time zone,
"last_sync_status_message" text,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
ALTER TABLE "archived_emails" ADD CONSTRAINT "archived_emails_custodian_id_custodians_id_fk" FOREIGN KEY ("custodian_id") REFERENCES "public"."custodians"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "email_attachments" ADD CONSTRAINT "email_attachments_email_id_archived_emails_id_fk" FOREIGN KEY ("email_id") REFERENCES "public"."archived_emails"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "email_attachments" ADD CONSTRAINT "email_attachments_attachment_id_attachments_id_fk" FOREIGN KEY ("attachment_id") REFERENCES "public"."attachments"("id") ON DELETE restrict ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "export_jobs" ADD CONSTRAINT "export_jobs_case_id_ediscovery_cases_id_fk" FOREIGN KEY ("case_id") REFERENCES "public"."ediscovery_cases"("id") ON DELETE set null ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "legal_holds" ADD CONSTRAINT "legal_holds_case_id_ediscovery_cases_id_fk" FOREIGN KEY ("case_id") REFERENCES "public"."ediscovery_cases"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "legal_holds" ADD CONSTRAINT "legal_holds_custodian_id_custodians_id_fk" FOREIGN KEY ("custodian_id") REFERENCES "public"."custodians"("id") ON DELETE cascade ON UPDATE no action;

View File

@@ -0,0 +1,818 @@
{
"id": "3fe238cc-60db-4ddb-8945-11db89bdee2b",
"prevId": "00000000-0000-0000-0000-000000000000",
"version": "7",
"dialect": "postgresql",
"tables": {
"public.archived_emails": {
"name": "archived_emails",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"custodian_id": {
"name": "custodian_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"message_id_header": {
"name": "message_id_header",
"type": "text",
"primaryKey": false,
"notNull": true
},
"sent_at": {
"name": "sent_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"subject": {
"name": "subject",
"type": "text",
"primaryKey": false,
"notNull": false
},
"sender_name": {
"name": "sender_name",
"type": "text",
"primaryKey": false,
"notNull": false
},
"sender_email": {
"name": "sender_email",
"type": "text",
"primaryKey": false,
"notNull": true
},
"recipients": {
"name": "recipients",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"storage_path": {
"name": "storage_path",
"type": "text",
"primaryKey": false,
"notNull": true
},
"storage_hash_sha256": {
"name": "storage_hash_sha256",
"type": "text",
"primaryKey": false,
"notNull": true
},
"size_bytes": {
"name": "size_bytes",
"type": "bigint",
"primaryKey": false,
"notNull": true
},
"is_indexed": {
"name": "is_indexed",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"has_attachments": {
"name": "has_attachments",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"is_on_legal_hold": {
"name": "is_on_legal_hold",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"archived_at": {
"name": "archived_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {
"archived_emails_custodian_id_custodians_id_fk": {
"name": "archived_emails_custodian_id_custodians_id_fk",
"tableFrom": "archived_emails",
"tableTo": "custodians",
"columnsFrom": [
"custodian_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.attachments": {
"name": "attachments",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"filename": {
"name": "filename",
"type": "text",
"primaryKey": false,
"notNull": true
},
"mime_type": {
"name": "mime_type",
"type": "text",
"primaryKey": false,
"notNull": false
},
"size_bytes": {
"name": "size_bytes",
"type": "bigint",
"primaryKey": false,
"notNull": true
},
"content_hash_sha256": {
"name": "content_hash_sha256",
"type": "text",
"primaryKey": false,
"notNull": true
},
"storage_path": {
"name": "storage_path",
"type": "text",
"primaryKey": false,
"notNull": true
}
},
"indexes": {},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {
"attachments_content_hash_sha256_unique": {
"name": "attachments_content_hash_sha256_unique",
"nullsNotDistinct": false,
"columns": [
"content_hash_sha256"
]
}
},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.email_attachments": {
"name": "email_attachments",
"schema": "",
"columns": {
"email_id": {
"name": "email_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"attachment_id": {
"name": "attachment_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
}
},
"indexes": {},
"foreignKeys": {
"email_attachments_email_id_archived_emails_id_fk": {
"name": "email_attachments_email_id_archived_emails_id_fk",
"tableFrom": "email_attachments",
"tableTo": "archived_emails",
"columnsFrom": [
"email_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"email_attachments_attachment_id_attachments_id_fk": {
"name": "email_attachments_attachment_id_attachments_id_fk",
"tableFrom": "email_attachments",
"tableTo": "attachments",
"columnsFrom": [
"attachment_id"
],
"columnsTo": [
"id"
],
"onDelete": "restrict",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {
"email_attachments_email_id_attachment_id_pk": {
"name": "email_attachments_email_id_attachment_id_pk",
"columns": [
"email_id",
"attachment_id"
]
}
},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.audit_logs": {
"name": "audit_logs",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "bigserial",
"primaryKey": true,
"notNull": true
},
"timestamp": {
"name": "timestamp",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"actor_identifier": {
"name": "actor_identifier",
"type": "text",
"primaryKey": false,
"notNull": true
},
"action": {
"name": "action",
"type": "text",
"primaryKey": false,
"notNull": true
},
"target_type": {
"name": "target_type",
"type": "text",
"primaryKey": false,
"notNull": false
},
"target_id": {
"name": "target_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"details": {
"name": "details",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"is_tamper_evident": {
"name": "is_tamper_evident",
"type": "boolean",
"primaryKey": false,
"notNull": false,
"default": false
}
},
"indexes": {},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.ediscovery_cases": {
"name": "ediscovery_cases",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"name": {
"name": "name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"description": {
"name": "description",
"type": "text",
"primaryKey": false,
"notNull": false
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'open'"
},
"created_by_identifier": {
"name": "created_by_identifier",
"type": "text",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {
"ediscovery_cases_name_unique": {
"name": "ediscovery_cases_name_unique",
"nullsNotDistinct": false,
"columns": [
"name"
]
}
},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.export_jobs": {
"name": "export_jobs",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"case_id": {
"name": "case_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"format": {
"name": "format",
"type": "text",
"primaryKey": false,
"notNull": true
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"query": {
"name": "query",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"file_path": {
"name": "file_path",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_by_identifier": {
"name": "created_by_identifier",
"type": "text",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"completed_at": {
"name": "completed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {},
"foreignKeys": {
"export_jobs_case_id_ediscovery_cases_id_fk": {
"name": "export_jobs_case_id_ediscovery_cases_id_fk",
"tableFrom": "export_jobs",
"tableTo": "ediscovery_cases",
"columnsFrom": [
"case_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.legal_holds": {
"name": "legal_holds",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"case_id": {
"name": "case_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"custodian_id": {
"name": "custodian_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"hold_criteria": {
"name": "hold_criteria",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"reason": {
"name": "reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"applied_by_identifier": {
"name": "applied_by_identifier",
"type": "text",
"primaryKey": false,
"notNull": true
},
"applied_at": {
"name": "applied_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"removed_at": {
"name": "removed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {},
"foreignKeys": {
"legal_holds_case_id_ediscovery_cases_id_fk": {
"name": "legal_holds_case_id_ediscovery_cases_id_fk",
"tableFrom": "legal_holds",
"tableTo": "ediscovery_cases",
"columnsFrom": [
"case_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"legal_holds_custodian_id_custodians_id_fk": {
"name": "legal_holds_custodian_id_custodians_id_fk",
"tableFrom": "legal_holds",
"tableTo": "custodians",
"columnsFrom": [
"custodian_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.retention_policies": {
"name": "retention_policies",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"name": {
"name": "name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"description": {
"name": "description",
"type": "text",
"primaryKey": false,
"notNull": false
},
"priority": {
"name": "priority",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"retention_period_days": {
"name": "retention_period_days",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"action_on_expiry": {
"name": "action_on_expiry",
"type": "retention_action",
"typeSchema": "public",
"primaryKey": false,
"notNull": true
},
"is_enabled": {
"name": "is_enabled",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": true
},
"conditions": {
"name": "conditions",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {
"retention_policies_name_unique": {
"name": "retention_policies_name_unique",
"nullsNotDistinct": false,
"columns": [
"name"
]
}
},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.custodians": {
"name": "custodians",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"email": {
"name": "email",
"type": "text",
"primaryKey": false,
"notNull": true
},
"display_name": {
"name": "display_name",
"type": "text",
"primaryKey": false,
"notNull": false
},
"source_type": {
"name": "source_type",
"type": "ingestion_provider",
"typeSchema": "public",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {
"custodians_email_unique": {
"name": "custodians_email_unique",
"nullsNotDistinct": false,
"columns": [
"email"
]
}
},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.ingestion_sources": {
"name": "ingestion_sources",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"name": {
"name": "name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"provider": {
"name": "provider",
"type": "ingestion_provider",
"typeSchema": "public",
"primaryKey": false,
"notNull": true
},
"credentials": {
"name": "credentials",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"status": {
"name": "status",
"type": "ingestion_status",
"typeSchema": "public",
"primaryKey": false,
"notNull": true,
"default": "'pending_auth'"
},
"last_sync_started_at": {
"name": "last_sync_started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"last_sync_finished_at": {
"name": "last_sync_finished_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"last_sync_status_message": {
"name": "last_sync_status_message",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
}
},
"enums": {
"public.retention_action": {
"name": "retention_action",
"schema": "public",
"values": [
"delete_permanently",
"notify_admin"
]
},
"public.ingestion_provider": {
"name": "ingestion_provider",
"schema": "public",
"values": [
"google_workspace",
"microsoft_365",
"generic_imap"
]
},
"public.ingestion_status": {
"name": "ingestion_status",
"schema": "public",
"values": [
"active",
"paused",
"error",
"pending_auth",
"syncing"
]
}
},
"schemas": {},
"sequences": {},
"roles": {},
"policies": {},
"views": {},
"_meta": {
"columns": {},
"schemas": {},
"tables": {}
}
}

View File

@@ -0,0 +1,13 @@
{
"version": "7",
"dialect": "postgresql",
"entries": [
{
"idx": 0,
"version": "7",
"when": 1752225352591,
"tag": "0000_amusing_namora",
"breakpoints": true
}
]
}

View File

@@ -0,0 +1,6 @@
export * from './schema/archived-emails';
export * from './schema/attachments';
export * from './schema/audit-logs';
export * from './schema/compliance';
export * from './schema/custodians';
export * from './schema/ingestion-sources';

View File

@@ -0,0 +1,28 @@
import { relations } from 'drizzle-orm';
import { boolean, jsonb, pgTable, text, timestamp, uuid, bigint } from 'drizzle-orm/pg-core';
import { custodians } from './custodians';
export const archivedEmails = pgTable('archived_emails', {
id: uuid('id').primaryKey().defaultRandom(),
custodianId: uuid('custodian_id').notNull().references(() => custodians.id, { onDelete: 'cascade' }),
messageIdHeader: text('message_id_header').notNull(),
sentAt: timestamp('sent_at', { withTimezone: true }).notNull(),
subject: text('subject'),
senderName: text('sender_name'),
senderEmail: text('sender_email').notNull(),
recipients: jsonb('recipients'),
storagePath: text('storage_path').notNull(),
storageHashSha256: text('storage_hash_sha256').notNull(),
sizeBytes: bigint('size_bytes', { mode: 'number' }).notNull(),
isIndexed: boolean('is_indexed').notNull().default(false),
hasAttachments: boolean('has_attachments').notNull().default(false),
isOnLegalHold: boolean('is_on_legal_hold').notNull().default(false),
archivedAt: timestamp('archived_at', { withTimezone: true }).notNull().defaultNow(),
});
export const archivedEmailsRelations = relations(archivedEmails, ({ one }) => ({
custodian: one(custodians, {
fields: [archivedEmails.custodianId],
references: [custodians.id],
}),
}));

View File

@@ -0,0 +1,34 @@
import { relations } from 'drizzle-orm';
import { pgTable, text, uuid, bigint, primaryKey } from 'drizzle-orm/pg-core';
import { archivedEmails } from './archived-emails';
export const attachments = pgTable('attachments', {
id: uuid('id').primaryKey().defaultRandom(),
filename: text('filename').notNull(),
mimeType: text('mime_type'),
sizeBytes: bigint('size_bytes', { mode: 'number' }).notNull(),
contentHashSha256: text('content_hash_sha256').notNull().unique(),
storagePath: text('storage_path').notNull(),
});
export const emailAttachments = pgTable('email_attachments', {
emailId: uuid('email_id').notNull().references(() => archivedEmails.id, { onDelete: 'cascade' }),
attachmentId: uuid('attachment_id').notNull().references(() => attachments.id, { onDelete: 'restrict' }),
}, (t) => ({
pk: primaryKey({ columns: [t.emailId, t.attachmentId] }),
}));
export const attachmentsRelations = relations(attachments, ({ many }) => ({
emailAttachments: many(emailAttachments),
}));
export const emailAttachmentsRelations = relations(emailAttachments, ({ one }) => ({
archivedEmail: one(archivedEmails, {
fields: [emailAttachments.emailId],
references: [archivedEmails.id],
}),
attachment: one(attachments, {
fields: [emailAttachments.attachmentId],
references: [attachments.id],
}),
}));

View File

@@ -0,0 +1,12 @@
import { bigserial, boolean, jsonb, pgTable, text, timestamp } from 'drizzle-orm/pg-core';
export const auditLogs = pgTable('audit_logs', {
id: bigserial('id', { mode: 'number' }).primaryKey(),
timestamp: timestamp('timestamp', { withTimezone: true }).notNull().defaultNow(),
actorIdentifier: text('actor_identifier').notNull(),
action: text('action').notNull(),
targetType: text('target_type'),
targetId: text('target_id'),
details: jsonb('details'),
isTamperEvident: boolean('is_tamper_evident').default(false),
});

View File

@@ -0,0 +1,80 @@
import { relations } from 'drizzle-orm';
import { boolean, integer, jsonb, pgEnum, pgTable, text, timestamp, uuid } from 'drizzle-orm/pg-core';
import { custodians } from './custodians';
// --- Enums ---
export const retentionActionEnum = pgEnum('retention_action', ['delete_permanently', 'notify_admin']);
// --- Tables ---
export const retentionPolicies = pgTable('retention_policies', {
id: uuid('id').primaryKey().defaultRandom(),
name: text('name').notNull().unique(),
description: text('description'),
priority: integer('priority').notNull(),
retentionPeriodDays: integer('retention_period_days').notNull(),
actionOnExpiry: retentionActionEnum('action_on_expiry').notNull(),
isEnabled: boolean('is_enabled').notNull().default(true),
conditions: jsonb('conditions'),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
});
export const ediscoveryCases = pgTable('ediscovery_cases', {
id: uuid('id').primaryKey().defaultRandom(),
name: text('name').notNull().unique(),
description: text('description'),
status: text('status').notNull().default('open'),
createdByIdentifier: text('created_by_identifier').notNull(),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
});
export const legalHolds = pgTable('legal_holds', {
id: uuid('id').primaryKey().defaultRandom(),
caseId: uuid('case_id').notNull().references(() => ediscoveryCases.id, { onDelete: 'cascade' }),
custodianId: uuid('custodian_id').references(() => custodians.id, { onDelete: 'cascade' }),
holdCriteria: jsonb('hold_criteria'),
reason: text('reason'),
appliedByIdentifier: text('applied_by_identifier').notNull(),
appliedAt: timestamp('applied_at', { withTimezone: true }).notNull().defaultNow(),
removedAt: timestamp('removed_at', { withTimezone: true }),
});
export const exportJobs = pgTable('export_jobs', {
id: uuid('id').primaryKey().defaultRandom(),
caseId: uuid('case_id').references(() => ediscoveryCases.id, { onDelete: 'set null' }),
format: text('format').notNull(),
status: text('status').notNull().default('pending'),
query: jsonb('query').notNull(),
filePath: text('file_path'),
createdByIdentifier: text('created_by_identifier').notNull(),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
completedAt: timestamp('completed_at', { withTimezone: true }),
});
// --- Relations ---
export const ediscoveryCasesRelations = relations(ediscoveryCases, ({ many }) => ({
legalHolds: many(legalHolds),
exportJobs: many(exportJobs),
}));
export const legalHoldsRelations = relations(legalHolds, ({ one }) => ({
ediscoveryCase: one(ediscoveryCases, {
fields: [legalHolds.caseId],
references: [ediscoveryCases.id],
}),
custodian: one(custodians, {
fields: [legalHolds.custodianId],
references: [custodians.id],
}),
}));
export const exportJobsRelations = relations(exportJobs, ({ one }) => ({
ediscoveryCase: one(ediscoveryCases, {
fields: [exportJobs.caseId],
references: [ediscoveryCases.id],
}),
}));

View File

@@ -0,0 +1,11 @@
import { pgTable, text, timestamp, uuid } from 'drizzle-orm/pg-core';
import { ingestionProviderEnum } from './ingestion-sources';
export const custodians = pgTable('custodians', {
id: uuid('id').primaryKey().defaultRandom(),
email: text('email').notNull().unique(),
displayName: text('display_name'),
sourceType: ingestionProviderEnum('source_type').notNull(),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow()
});

View File

@@ -0,0 +1,28 @@
import { jsonb, pgEnum, pgTable, text, timestamp, uuid } from 'drizzle-orm/pg-core';
export const ingestionProviderEnum = pgEnum('ingestion_provider', [
'google_workspace',
'microsoft_365',
'generic_imap'
]);
export const ingestionStatusEnum = pgEnum('ingestion_status', [
'active',
'paused',
'error',
'pending_auth',
'syncing'
]);
export const ingestionSources = pgTable('ingestion_sources', {
id: uuid('id').primaryKey().defaultRandom(),
name: text('name').notNull(),
provider: ingestionProviderEnum('provider').notNull(),
credentials: jsonb('credentials'),
status: ingestionStatusEnum('status').notNull().default('pending_auth'),
lastSyncStartedAt: timestamp('last_sync_started_at', { withTimezone: true }),
lastSyncFinishedAt: timestamp('last_sync_finished_at', { withTimezone: true }),
lastSyncStatusMessage: text('last_sync_status_message'),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow()
});

View File

@@ -1,8 +1,10 @@
import express from 'express';
import dotenv from 'dotenv';
import { AuthController } from './api/controllers/auth.controller';
import { IngestionController } from './api/controllers/ingestion.controller';
import { requireAuth } from './api/middleware/requireAuth';
import { createAuthRouter } from './api/routes/auth.routes';
import { createIngestionRouter } from './api/routes/ingestion.routes';
import { AuthService } from './services/AuthService';
import { AdminUserService } from './services/UserService';
@@ -27,6 +29,7 @@ if (!PORT_BACKEND || !JWT_SECRET || !JWT_EXPIRES_IN) {
const userService = new AdminUserService();
const authService = new AuthService(userService, JWT_SECRET, JWT_EXPIRES_IN);
const authController = new AuthController(authService);
const ingestionController = new IngestionController();
// --- Express App Initialization ---
const app = express();
@@ -36,13 +39,15 @@ app.use(express.json()); // For parsing application/json
// --- Routes ---
const authRouter = createAuthRouter(authController);
const ingestionRouter = createIngestionRouter(ingestionController, authService);
app.use('/v1/auth', authRouter);
app.use('/v1/ingestion-sources', ingestionRouter);
// Example of a protected route
app.get('/v1/protected', requireAuth(authService), (req, res) => {
res.json({
message: 'You have accessed a protected route!',
user: req.user, // The user payload is attached by the requireAuth middleware
user: req.user // The user payload is attached by the requireAuth middleware
});
});

View File

@@ -0,0 +1,85 @@
import { db } from '../database';
import { ingestionSources } from '../database/schema';
import type { CreateIngestionSourceDto, UpdateIngestionSourceDto } from '@open-archive/types';
import { eq } from 'drizzle-orm';
import { Queue } from 'bullmq';
// This assumes you have a BullMQ queue instance exported from somewhere
// In a real setup, this would be injected or imported from a central place.
let initialImportQueue: Queue;
// TODO: Initialize and connect to the actual BullMQ queue.
// For now, we'll use a mock for demonstration purposes.
if (process.env.NODE_ENV !== 'production') {
initialImportQueue = {
add: async (name: string, data: any) => {
console.log(`Mock Queue: Job '${name}' added with data:`, data);
return Promise.resolve({} as any);
}
} as Queue;
}
export class IngestionService {
public static async create(dto: CreateIngestionSourceDto) {
// The DTO from the frontend uses `providerConfig` for simplicity.
// We map it to the `credentials` column in the database schema.
const { providerConfig, ...rest } = dto;
const valuesToInsert = {
...rest,
credentials: providerConfig
};
const [newSource] = await db.insert(ingestionSources).values(valuesToInsert).returning();
return newSource;
}
public static async findAll() {
return await db.select().from(ingestionSources);
}
public static async findById(id: string) {
const [source] = await db.select().from(ingestionSources).where(eq(ingestionSources.id, id));
if (!source) {
throw new Error('Ingestion source not found');
}
return source;
}
public static async update(id: string, dto: UpdateIngestionSourceDto) {
// The DTO from the frontend uses `providerConfig` for simplicity.
// We map it to the `credentials` column in the database schema if it exists.
const { providerConfig, ...rest } = dto;
const valuesToUpdate: Partial<typeof ingestionSources.$inferInsert> = { ...rest };
if (providerConfig) {
valuesToUpdate.credentials = providerConfig;
}
const [updatedSource] = await db
.update(ingestionSources)
.set(valuesToUpdate)
.where(eq(ingestionSources.id, id))
.returning();
if (!updatedSource) {
throw new Error('Ingestion source not found');
}
return updatedSource;
}
public static async delete(id: string) {
const [deletedSource] = await db
.delete(ingestionSources)
.where(eq(ingestionSources.id, id))
.returning();
if (!deletedSource) {
throw new Error('Ingestion source not found');
}
return deletedSource;
}
public static async triggerInitialImport(id: string) {
const source = await this.findById(id);
await initialImportQueue.add('initial-import', { ingestionSourceId: source.id });
return await this.update(id, { status: 'syncing' });
}
}

View File

@@ -14,11 +14,12 @@
"lint": "prettier --check ."
},
"dependencies": {
"@open-archive/types": "workspace:*"
"@open-archive/types": "workspace:*",
"lucide-svelte": "^0.525.0"
},
"devDependencies": {
"@internationalized/date": "^3.8.2",
"@lucide/svelte": "^0.525.0",
"@lucide/svelte": "^0.515.0",
"@sveltejs/adapter-auto": "^6.0.0",
"@sveltejs/kit": "^2.16.0",
"@sveltejs/vite-plugin-svelte": "^5.0.0",

View File

@@ -9,11 +9,17 @@ const BASE_URL = '/api/v1'; // Using a relative URL for proxying
* @param options The standard Fetch API options.
* @returns A Promise that resolves to the Fetch Response.
*/
export const api = async (url: string, options: RequestInit = {}): Promise<Response> => {
type Fetch = typeof fetch;
export const api = async (
url: string,
options: RequestInit = {},
customFetch: Fetch = fetch
): Promise<Response> => {
const { accessToken } = get(authStore);
const defaultHeaders: HeadersInit = {
'Content-Type': 'application/json',
'Content-Type': 'application/json'
};
if (accessToken) {
@@ -24,9 +30,9 @@ export const api = async (url: string, options: RequestInit = {}): Promise<Respo
...options,
headers: {
...defaultHeaders,
...options.headers,
},
...options.headers
}
};
return fetch(`${BASE_URL}${url}`, mergedOptions);
return customFetch(`${BASE_URL}${url}`, mergedOptions);
};

View File

@@ -0,0 +1,101 @@
<script lang="ts">
import type { IngestionSource, CreateIngestionSourceDto } from '@open-archive/types';
import { Button } from '$lib/components/ui/button';
import * as Dialog from '$lib/components/ui/dialog';
import { Input } from '$lib/components/ui/input';
import { Label } from '$lib/components/ui/label';
import * as Select from '$lib/components/ui/select';
let {
source = null,
onSubmit
}: {
source?: IngestionSource | null;
onSubmit: (data: CreateIngestionSourceDto) => void;
} = $props();
const providerOptions = [
{ value: 'google_workspace', label: 'Google Workspace' },
{ value: 'microsoft_365', label: 'Microsoft 365' },
{ value: 'generic_imap', label: 'Generic IMAP' }
];
let formData = $state({
name: source?.name ?? '',
provider: source?.provider ?? 'google_workspace',
providerConfig: source?.providerConfig ?? {}
});
const triggerContent = $derived(
providerOptions.find((p) => p.value === formData.provider)?.label ?? 'Select a provider'
);
const handleSubmit = (event: Event) => {
event.preventDefault();
onSubmit(formData);
};
</script>
<form onsubmit={handleSubmit} class="grid gap-4 py-4">
<div class="grid grid-cols-4 items-center gap-4">
<Label for="name" class="text-right">Name</Label>
<Input id="name" bind:value={formData.name} class="col-span-3" />
</div>
<div class="grid grid-cols-4 items-center gap-4">
<Label for="provider" class="text-right">Provider</Label>
<Select.Root name="provider" bind:value={formData.provider} type="single">
<Select.Trigger class="col-span-3">
{triggerContent}
</Select.Trigger>
<Select.Content>
{#each providerOptions as option}
<Select.Item value={option.value}>{option.label}</Select.Item>
{/each}
</Select.Content>
</Select.Root>
</div>
{#if formData.provider === 'google_workspace' || formData.provider === 'microsoft_365'}
<div class="grid grid-cols-4 items-center gap-4">
<Label for="clientId" class="text-right">Client ID</Label>
<Input id="clientId" bind:value={formData.providerConfig.clientId} class="col-span-3" />
</div>
<div class="grid grid-cols-4 items-center gap-4">
<Label for="clientSecret" class="text-right">Client Secret</Label>
<Input
id="clientSecret"
bind:value={formData.providerConfig.clientSecret}
class="col-span-3"
/>
</div>
<div class="grid grid-cols-4 items-center gap-4">
<Label for="redirectUri" class="text-right">Redirect URI</Label>
<Input id="redirectUri" bind:value={formData.providerConfig.redirectUri} class="col-span-3" />
</div>
{:else if formData.provider === 'generic_imap'}
<div class="grid grid-cols-4 items-center gap-4">
<Label for="host" class="text-right">Host</Label>
<Input id="host" bind:value={formData.providerConfig.host} class="col-span-3" />
</div>
<div class="grid grid-cols-4 items-center gap-4">
<Label for="port" class="text-right">Port</Label>
<Input id="port" type="number" bind:value={formData.providerConfig.port} class="col-span-3" />
</div>
<div class="grid grid-cols-4 items-center gap-4">
<Label for="username" class="text-right">Username</Label>
<Input id="username" bind:value={formData.providerConfig.username} class="col-span-3" />
</div>
<div class="grid grid-cols-4 items-center gap-4">
<Label for="password" class="text-right">Password</Label>
<Input
id="password"
type="password"
bind:value={formData.providerConfig.password}
class="col-span-3"
/>
</div>
{/if}
<Dialog.Footer>
<Button type="submit">Save changes</Button>
</Dialog.Footer>
</form>

View File

@@ -0,0 +1,7 @@
<script lang="ts">
import { Dialog as DialogPrimitive } from "bits-ui";
let { ref = $bindable(null), ...restProps }: DialogPrimitive.CloseProps = $props();
</script>
<DialogPrimitive.Close bind:ref data-slot="dialog-close" {...restProps} />

View File

@@ -0,0 +1,43 @@
<script lang="ts">
import { Dialog as DialogPrimitive } from "bits-ui";
import XIcon from "@lucide/svelte/icons/x";
import type { Snippet } from "svelte";
import * as Dialog from "./index.js";
import { cn, type WithoutChildrenOrChild } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
portalProps,
children,
showCloseButton = true,
...restProps
}: WithoutChildrenOrChild<DialogPrimitive.ContentProps> & {
portalProps?: DialogPrimitive.PortalProps;
children: Snippet;
showCloseButton?: boolean;
} = $props();
</script>
<Dialog.Portal {...portalProps}>
<Dialog.Overlay />
<DialogPrimitive.Content
bind:ref
data-slot="dialog-content"
class={cn(
"bg-background data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 fixed left-[50%] top-[50%] z-50 grid w-full max-w-[calc(100%-2rem)] translate-x-[-50%] translate-y-[-50%] gap-4 rounded-lg border p-6 shadow-lg duration-200 sm:max-w-lg",
className
)}
{...restProps}
>
{@render children?.()}
{#if showCloseButton}
<DialogPrimitive.Close
class="ring-offset-background focus:ring-ring rounded-xs focus:outline-hidden absolute right-4 top-4 opacity-70 transition-opacity hover:opacity-100 focus:ring-2 focus:ring-offset-2 disabled:pointer-events-none [&_svg:not([class*='size-'])]:size-4 [&_svg]:pointer-events-none [&_svg]:shrink-0"
>
<XIcon />
<span class="sr-only">Close</span>
</DialogPrimitive.Close>
{/if}
</DialogPrimitive.Content>
</Dialog.Portal>

View File

@@ -0,0 +1,17 @@
<script lang="ts">
import { Dialog as DialogPrimitive } from "bits-ui";
import { cn } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
...restProps
}: DialogPrimitive.DescriptionProps = $props();
</script>
<DialogPrimitive.Description
bind:ref
data-slot="dialog-description"
class={cn("text-muted-foreground text-sm", className)}
{...restProps}
/>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLDivElement>> = $props();
</script>
<div
bind:this={ref}
data-slot="dialog-footer"
class={cn("flex flex-col-reverse gap-2 sm:flex-row sm:justify-end", className)}
{...restProps}
>
{@render children?.()}
</div>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import type { HTMLAttributes } from "svelte/elements";
import { cn, type WithElementRef } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLDivElement>> = $props();
</script>
<div
bind:this={ref}
data-slot="dialog-header"
class={cn("flex flex-col gap-2 text-center sm:text-left", className)}
{...restProps}
>
{@render children?.()}
</div>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import { Dialog as DialogPrimitive } from "bits-ui";
import { cn } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
...restProps
}: DialogPrimitive.OverlayProps = $props();
</script>
<DialogPrimitive.Overlay
bind:ref
data-slot="dialog-overlay"
class={cn(
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 fixed inset-0 z-50 bg-black/50",
className
)}
{...restProps}
/>

View File

@@ -0,0 +1,17 @@
<script lang="ts">
import { Dialog as DialogPrimitive } from "bits-ui";
import { cn } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
...restProps
}: DialogPrimitive.TitleProps = $props();
</script>
<DialogPrimitive.Title
bind:ref
data-slot="dialog-title"
class={cn("text-lg font-semibold leading-none", className)}
{...restProps}
/>

View File

@@ -0,0 +1,7 @@
<script lang="ts">
import { Dialog as DialogPrimitive } from "bits-ui";
let { ref = $bindable(null), ...restProps }: DialogPrimitive.TriggerProps = $props();
</script>
<DialogPrimitive.Trigger bind:ref data-slot="dialog-trigger" {...restProps} />

View File

@@ -0,0 +1,37 @@
import { Dialog as DialogPrimitive } from "bits-ui";
import Title from "./dialog-title.svelte";
import Footer from "./dialog-footer.svelte";
import Header from "./dialog-header.svelte";
import Overlay from "./dialog-overlay.svelte";
import Content from "./dialog-content.svelte";
import Description from "./dialog-description.svelte";
import Trigger from "./dialog-trigger.svelte";
import Close from "./dialog-close.svelte";
const Root = DialogPrimitive.Root;
const Portal = DialogPrimitive.Portal;
export {
Root,
Title,
Portal,
Footer,
Header,
Trigger,
Overlay,
Content,
Description,
Close,
//
Root as Dialog,
Title as DialogTitle,
Portal as DialogPortal,
Footer as DialogFooter,
Header as DialogHeader,
Trigger as DialogTrigger,
Overlay as DialogOverlay,
Content as DialogContent,
Description as DialogDescription,
Close as DialogClose,
};

View File

@@ -0,0 +1,41 @@
<script lang="ts">
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
import CheckIcon from "@lucide/svelte/icons/check";
import MinusIcon from "@lucide/svelte/icons/minus";
import { cn, type WithoutChildrenOrChild } from "$lib/utils.js";
import type { Snippet } from "svelte";
let {
ref = $bindable(null),
checked = $bindable(false),
indeterminate = $bindable(false),
class: className,
children: childrenProp,
...restProps
}: WithoutChildrenOrChild<DropdownMenuPrimitive.CheckboxItemProps> & {
children?: Snippet;
} = $props();
</script>
<DropdownMenuPrimitive.CheckboxItem
bind:ref
bind:checked
bind:indeterminate
data-slot="dropdown-menu-checkbox-item"
class={cn(
"focus:bg-accent focus:text-accent-foreground outline-hidden relative flex cursor-default select-none items-center gap-2 rounded-sm py-1.5 pl-8 pr-2 text-sm data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg:not([class*='size-'])]:size-4 [&_svg]:pointer-events-none [&_svg]:shrink-0",
className
)}
{...restProps}
>
{#snippet children({ checked, indeterminate })}
<span class="pointer-events-none absolute left-2 flex size-3.5 items-center justify-center">
{#if indeterminate}
<MinusIcon class="size-4" />
{:else}
<CheckIcon class={cn("size-4", !checked && "text-transparent")} />
{/if}
</span>
{@render childrenProp?.()}
{/snippet}
</DropdownMenuPrimitive.CheckboxItem>

View File

@@ -0,0 +1,27 @@
<script lang="ts">
import { cn } from "$lib/utils.js";
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
let {
ref = $bindable(null),
sideOffset = 4,
portalProps,
class: className,
...restProps
}: DropdownMenuPrimitive.ContentProps & {
portalProps?: DropdownMenuPrimitive.PortalProps;
} = $props();
</script>
<DropdownMenuPrimitive.Portal {...portalProps}>
<DropdownMenuPrimitive.Content
bind:ref
data-slot="dropdown-menu-content"
{sideOffset}
class={cn(
"bg-popover text-popover-foreground data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 max-h-(--radix-dropdown-menu-content-available-height) origin-(--radix-dropdown-menu-content-transform-origin) z-50 min-w-[8rem] overflow-y-auto overflow-x-hidden rounded-md border p-1 shadow-md",
className
)}
{...restProps}
/>
</DropdownMenuPrimitive.Portal>

View File

@@ -0,0 +1,22 @@
<script lang="ts">
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
import { cn } from "$lib/utils.js";
import type { ComponentProps } from "svelte";
let {
ref = $bindable(null),
class: className,
inset,
...restProps
}: ComponentProps<typeof DropdownMenuPrimitive.GroupHeading> & {
inset?: boolean;
} = $props();
</script>
<DropdownMenuPrimitive.GroupHeading
bind:ref
data-slot="dropdown-menu-group-heading"
data-inset={inset}
class={cn("px-2 py-1.5 text-sm font-semibold data-[inset]:pl-8", className)}
{...restProps}
/>

View File

@@ -0,0 +1,7 @@
<script lang="ts">
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
let { ref = $bindable(null), ...restProps }: DropdownMenuPrimitive.GroupProps = $props();
</script>
<DropdownMenuPrimitive.Group bind:ref data-slot="dropdown-menu-group" {...restProps} />

View File

@@ -0,0 +1,27 @@
<script lang="ts">
import { cn } from "$lib/utils.js";
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
let {
ref = $bindable(null),
class: className,
inset,
variant = "default",
...restProps
}: DropdownMenuPrimitive.ItemProps & {
inset?: boolean;
variant?: "default" | "destructive";
} = $props();
</script>
<DropdownMenuPrimitive.Item
bind:ref
data-slot="dropdown-menu-item"
data-inset={inset}
data-variant={variant}
class={cn(
"data-highlighted:bg-accent data-highlighted:text-accent-foreground data-[variant=destructive]:text-destructive data-[variant=destructive]:data-highlighted:bg-destructive/10 dark:data-[variant=destructive]:data-highlighted:bg-destructive/20 data-[variant=destructive]:data-highlighted:text-destructive data-[variant=destructive]:*:[svg]:!text-destructive [&_svg:not([class*='text-'])]:text-muted-foreground outline-hidden relative flex cursor-default select-none items-center gap-2 rounded-sm px-2 py-1.5 text-sm data-[disabled]:pointer-events-none data-[inset]:pl-8 data-[disabled]:opacity-50 [&_svg:not([class*='size-'])]:size-4 [&_svg]:pointer-events-none [&_svg]:shrink-0",
className
)}
{...restProps}
/>

View File

@@ -0,0 +1,24 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
inset,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLDivElement>> & {
inset?: boolean;
} = $props();
</script>
<div
bind:this={ref}
data-slot="dropdown-menu-label"
data-inset={inset}
class={cn("px-2 py-1.5 text-sm font-semibold data-[inset]:pl-8", className)}
{...restProps}
>
{@render children?.()}
</div>

View File

@@ -0,0 +1,16 @@
<script lang="ts">
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
let {
ref = $bindable(null),
value = $bindable(),
...restProps
}: DropdownMenuPrimitive.RadioGroupProps = $props();
</script>
<DropdownMenuPrimitive.RadioGroup
bind:ref
bind:value
data-slot="dropdown-menu-radio-group"
{...restProps}
/>

View File

@@ -0,0 +1,31 @@
<script lang="ts">
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
import CircleIcon from "@lucide/svelte/icons/circle";
import { cn, type WithoutChild } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
children: childrenProp,
...restProps
}: WithoutChild<DropdownMenuPrimitive.RadioItemProps> = $props();
</script>
<DropdownMenuPrimitive.RadioItem
bind:ref
data-slot="dropdown-menu-radio-item"
class={cn(
"focus:bg-accent focus:text-accent-foreground outline-hidden relative flex cursor-default select-none items-center gap-2 rounded-sm py-1.5 pl-8 pr-2 text-sm data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg:not([class*='size-'])]:size-4 [&_svg]:pointer-events-none [&_svg]:shrink-0",
className
)}
{...restProps}
>
{#snippet children({ checked })}
<span class="pointer-events-none absolute left-2 flex size-3.5 items-center justify-center">
{#if checked}
<CircleIcon class="size-2 fill-current" />
{/if}
</span>
{@render childrenProp?.({ checked })}
{/snippet}
</DropdownMenuPrimitive.RadioItem>

View File

@@ -0,0 +1,17 @@
<script lang="ts">
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
import { cn } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
...restProps
}: DropdownMenuPrimitive.SeparatorProps = $props();
</script>
<DropdownMenuPrimitive.Separator
bind:ref
data-slot="dropdown-menu-separator"
class={cn("bg-border -mx-1 my-1 h-px", className)}
{...restProps}
/>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import type { HTMLAttributes } from "svelte/elements";
import { cn, type WithElementRef } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLSpanElement>> = $props();
</script>
<span
bind:this={ref}
data-slot="dropdown-menu-shortcut"
class={cn("text-muted-foreground ml-auto text-xs tracking-widest", className)}
{...restProps}
>
{@render children?.()}
</span>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
import { cn } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
...restProps
}: DropdownMenuPrimitive.SubContentProps = $props();
</script>
<DropdownMenuPrimitive.SubContent
bind:ref
data-slot="dropdown-menu-sub-content"
class={cn(
"bg-popover text-popover-foreground data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 origin-(--radix-dropdown-menu-content-transform-origin) z-50 min-w-[8rem] overflow-hidden rounded-md border p-1 shadow-lg",
className
)}
{...restProps}
/>

View File

@@ -0,0 +1,29 @@
<script lang="ts">
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
import ChevronRightIcon from "@lucide/svelte/icons/chevron-right";
import { cn } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
inset,
children,
...restProps
}: DropdownMenuPrimitive.SubTriggerProps & {
inset?: boolean;
} = $props();
</script>
<DropdownMenuPrimitive.SubTrigger
bind:ref
data-slot="dropdown-menu-sub-trigger"
data-inset={inset}
class={cn(
"data-highlighted:bg-accent data-highlighted:text-accent-foreground data-[state=open]:bg-accent data-[state=open]:text-accent-foreground outline-hidden [&_svg:not([class*='text-'])]:text-muted-foreground flex cursor-default select-none items-center gap-2 rounded-sm px-2 py-1.5 text-sm data-[disabled]:pointer-events-none data-[inset]:pl-8 data-[disabled]:opacity-50 [&_svg:not([class*='size-'])]:size-4 [&_svg]:pointer-events-none [&_svg]:shrink-0",
className
)}
{...restProps}
>
{@render children?.()}
<ChevronRightIcon class="ml-auto size-4" />
</DropdownMenuPrimitive.SubTrigger>

View File

@@ -0,0 +1,7 @@
<script lang="ts">
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
let { ref = $bindable(null), ...restProps }: DropdownMenuPrimitive.TriggerProps = $props();
</script>
<DropdownMenuPrimitive.Trigger bind:ref data-slot="dropdown-menu-trigger" {...restProps} />

View File

@@ -0,0 +1,49 @@
import { DropdownMenu as DropdownMenuPrimitive } from "bits-ui";
import CheckboxItem from "./dropdown-menu-checkbox-item.svelte";
import Content from "./dropdown-menu-content.svelte";
import Group from "./dropdown-menu-group.svelte";
import Item from "./dropdown-menu-item.svelte";
import Label from "./dropdown-menu-label.svelte";
import RadioGroup from "./dropdown-menu-radio-group.svelte";
import RadioItem from "./dropdown-menu-radio-item.svelte";
import Separator from "./dropdown-menu-separator.svelte";
import Shortcut from "./dropdown-menu-shortcut.svelte";
import Trigger from "./dropdown-menu-trigger.svelte";
import SubContent from "./dropdown-menu-sub-content.svelte";
import SubTrigger from "./dropdown-menu-sub-trigger.svelte";
import GroupHeading from "./dropdown-menu-group-heading.svelte";
const Sub = DropdownMenuPrimitive.Sub;
const Root = DropdownMenuPrimitive.Root;
export {
CheckboxItem,
Content,
Root as DropdownMenu,
CheckboxItem as DropdownMenuCheckboxItem,
Content as DropdownMenuContent,
Group as DropdownMenuGroup,
Item as DropdownMenuItem,
Label as DropdownMenuLabel,
RadioGroup as DropdownMenuRadioGroup,
RadioItem as DropdownMenuRadioItem,
Separator as DropdownMenuSeparator,
Shortcut as DropdownMenuShortcut,
Sub as DropdownMenuSub,
SubContent as DropdownMenuSubContent,
SubTrigger as DropdownMenuSubTrigger,
Trigger as DropdownMenuTrigger,
GroupHeading as DropdownMenuGroupHeading,
Group,
GroupHeading,
Item,
Label,
RadioGroup,
RadioItem,
Root,
Separator,
Shortcut,
Sub,
SubContent,
SubTrigger,
Trigger,
};

View File

@@ -0,0 +1,37 @@
import { Select as SelectPrimitive } from "bits-ui";
import Group from "./select-group.svelte";
import Label from "./select-label.svelte";
import Item from "./select-item.svelte";
import Content from "./select-content.svelte";
import Trigger from "./select-trigger.svelte";
import Separator from "./select-separator.svelte";
import ScrollDownButton from "./select-scroll-down-button.svelte";
import ScrollUpButton from "./select-scroll-up-button.svelte";
import GroupHeading from "./select-group-heading.svelte";
const Root = SelectPrimitive.Root;
export {
Root,
Group,
Label,
Item,
Content,
Trigger,
Separator,
ScrollDownButton,
ScrollUpButton,
GroupHeading,
//
Root as Select,
Group as SelectGroup,
Label as SelectLabel,
Item as SelectItem,
Content as SelectContent,
Trigger as SelectTrigger,
Separator as SelectSeparator,
ScrollDownButton as SelectScrollDownButton,
ScrollUpButton as SelectScrollUpButton,
GroupHeading as SelectGroupHeading,
};

View File

@@ -0,0 +1,40 @@
<script lang="ts">
import { Select as SelectPrimitive } from "bits-ui";
import SelectScrollUpButton from "./select-scroll-up-button.svelte";
import SelectScrollDownButton from "./select-scroll-down-button.svelte";
import { cn, type WithoutChild } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
sideOffset = 4,
portalProps,
children,
...restProps
}: WithoutChild<SelectPrimitive.ContentProps> & {
portalProps?: SelectPrimitive.PortalProps;
} = $props();
</script>
<SelectPrimitive.Portal {...portalProps}>
<SelectPrimitive.Content
bind:ref
{sideOffset}
data-slot="select-content"
class={cn(
"bg-popover text-popover-foreground data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 max-h-(--bits-select-content-available-height) origin-(--bits-select-content-transform-origin) relative z-50 min-w-[8rem] overflow-y-auto overflow-x-hidden rounded-md border shadow-md data-[side=bottom]:translate-y-1 data-[side=left]:-translate-x-1 data-[side=right]:translate-x-1 data-[side=top]:-translate-y-1",
className
)}
{...restProps}
>
<SelectScrollUpButton />
<SelectPrimitive.Viewport
class={cn(
"h-(--bits-select-anchor-height) min-w-(--bits-select-anchor-width) w-full scroll-my-1 p-1"
)}
>
{@render children?.()}
</SelectPrimitive.Viewport>
<SelectScrollDownButton />
</SelectPrimitive.Content>
</SelectPrimitive.Portal>

View File

@@ -0,0 +1,21 @@
<script lang="ts">
import { Select as SelectPrimitive } from "bits-ui";
import { cn } from "$lib/utils.js";
import type { ComponentProps } from "svelte";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: ComponentProps<typeof SelectPrimitive.GroupHeading> = $props();
</script>
<SelectPrimitive.GroupHeading
bind:ref
data-slot="select-group-heading"
class={cn("text-muted-foreground px-2 py-1.5 text-xs", className)}
{...restProps}
>
{@render children?.()}
</SelectPrimitive.GroupHeading>

View File

@@ -0,0 +1,7 @@
<script lang="ts">
import { Select as SelectPrimitive } from "bits-ui";
let { ref = $bindable(null), ...restProps }: SelectPrimitive.GroupProps = $props();
</script>
<SelectPrimitive.Group data-slot="select-group" {...restProps} />

View File

@@ -0,0 +1,38 @@
<script lang="ts">
import CheckIcon from "@lucide/svelte/icons/check";
import { Select as SelectPrimitive } from "bits-ui";
import { cn, type WithoutChild } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
value,
label,
children: childrenProp,
...restProps
}: WithoutChild<SelectPrimitive.ItemProps> = $props();
</script>
<SelectPrimitive.Item
bind:ref
{value}
data-slot="select-item"
class={cn(
"data-[highlighted]:bg-accent data-[highlighted]:text-accent-foreground [&_svg:not([class*='text-'])]:text-muted-foreground outline-hidden *:[span]:last:flex *:[span]:last:items-center *:[span]:last:gap-2 relative flex w-full cursor-default select-none items-center gap-2 rounded-sm py-1.5 pl-2 pr-8 text-sm data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg:not([class*='size-'])]:size-4 [&_svg]:pointer-events-none [&_svg]:shrink-0",
className
)}
{...restProps}
>
{#snippet children({ selected, highlighted })}
<span class="absolute right-2 flex size-3.5 items-center justify-center">
{#if selected}
<CheckIcon class="size-4" />
{/if}
</span>
{#if childrenProp}
{@render childrenProp({ selected, highlighted })}
{:else}
{label || value}
{/if}
{/snippet}
</SelectPrimitive.Item>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLDivElement>> & {} = $props();
</script>
<div
bind:this={ref}
data-slot="select-label"
class={cn("text-muted-foreground px-2 py-1.5 text-xs", className)}
{...restProps}
>
{@render children?.()}
</div>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import ChevronDownIcon from "@lucide/svelte/icons/chevron-down";
import { Select as SelectPrimitive } from "bits-ui";
import { cn, type WithoutChildrenOrChild } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
...restProps
}: WithoutChildrenOrChild<SelectPrimitive.ScrollDownButtonProps> = $props();
</script>
<SelectPrimitive.ScrollDownButton
bind:ref
data-slot="select-scroll-down-button"
class={cn("flex cursor-default items-center justify-center py-1", className)}
{...restProps}
>
<ChevronDownIcon class="size-4" />
</SelectPrimitive.ScrollDownButton>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import ChevronUpIcon from "@lucide/svelte/icons/chevron-up";
import { Select as SelectPrimitive } from "bits-ui";
import { cn, type WithoutChildrenOrChild } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
...restProps
}: WithoutChildrenOrChild<SelectPrimitive.ScrollUpButtonProps> = $props();
</script>
<SelectPrimitive.ScrollUpButton
bind:ref
data-slot="select-scroll-up-button"
class={cn("flex cursor-default items-center justify-center py-1", className)}
{...restProps}
>
<ChevronUpIcon class="size-4" />
</SelectPrimitive.ScrollUpButton>

View File

@@ -0,0 +1,18 @@
<script lang="ts">
import type { Separator as SeparatorPrimitive } from "bits-ui";
import { Separator } from "$lib/components/ui/separator/index.js";
import { cn } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
...restProps
}: SeparatorPrimitive.RootProps = $props();
</script>
<Separator
bind:ref
data-slot="select-separator"
class={cn("bg-border pointer-events-none -mx-1 my-1 h-px", className)}
{...restProps}
/>

View File

@@ -0,0 +1,29 @@
<script lang="ts">
import { Select as SelectPrimitive } from "bits-ui";
import ChevronDownIcon from "@lucide/svelte/icons/chevron-down";
import { cn, type WithoutChild } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
children,
size = "default",
...restProps
}: WithoutChild<SelectPrimitive.TriggerProps> & {
size?: "sm" | "default";
} = $props();
</script>
<SelectPrimitive.Trigger
bind:ref
data-slot="select-trigger"
data-size={size}
class={cn(
"border-input data-[placeholder]:text-muted-foreground [&_svg:not([class*='text-'])]:text-muted-foreground focus-visible:border-ring focus-visible:ring-ring/50 aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive dark:bg-input/30 dark:hover:bg-input/50 shadow-xs flex w-fit select-none items-center justify-between gap-2 whitespace-nowrap rounded-md border bg-transparent px-3 py-2 text-sm outline-none transition-[color,box-shadow] focus-visible:ring-[3px] disabled:cursor-not-allowed disabled:opacity-50 data-[size=default]:h-9 data-[size=sm]:h-8 *:data-[slot=select-value]:line-clamp-1 *:data-[slot=select-value]:flex *:data-[slot=select-value]:items-center *:data-[slot=select-value]:gap-2 [&_svg:not([class*='size-'])]:size-4 [&_svg]:pointer-events-none [&_svg]:shrink-0",
className
)}
{...restProps}
>
{@render children?.()}
<ChevronDownIcon class="size-4 opacity-50" />
</SelectPrimitive.Trigger>

View File

@@ -0,0 +1,7 @@
import Root from "./separator.svelte";
export {
Root,
//
Root as Separator,
};

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import { Separator as SeparatorPrimitive } from "bits-ui";
import { cn } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
...restProps
}: SeparatorPrimitive.RootProps = $props();
</script>
<SeparatorPrimitive.Root
bind:ref
data-slot="separator"
class={cn(
"bg-border shrink-0 data-[orientation=horizontal]:h-px data-[orientation=vertical]:h-full data-[orientation=horizontal]:w-full data-[orientation=vertical]:w-px",
className
)}
{...restProps}
/>

View File

@@ -0,0 +1,28 @@
import Root from "./table.svelte";
import Body from "./table-body.svelte";
import Caption from "./table-caption.svelte";
import Cell from "./table-cell.svelte";
import Footer from "./table-footer.svelte";
import Head from "./table-head.svelte";
import Header from "./table-header.svelte";
import Row from "./table-row.svelte";
export {
Root,
Body,
Caption,
Cell,
Footer,
Head,
Header,
Row,
//
Root as Table,
Body as TableBody,
Caption as TableCaption,
Cell as TableCell,
Footer as TableFooter,
Head as TableHead,
Header as TableHeader,
Row as TableRow,
};

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLTableSectionElement>> = $props();
</script>
<tbody
bind:this={ref}
data-slot="table-body"
class={cn("[&_tr:last-child]:border-0", className)}
{...restProps}
>
{@render children?.()}
</tbody>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLElement>> = $props();
</script>
<caption
bind:this={ref}
data-slot="table-caption"
class={cn("text-muted-foreground mt-4 text-sm", className)}
{...restProps}
>
{@render children?.()}
</caption>

View File

@@ -0,0 +1,23 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLTdAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLTdAttributes> = $props();
</script>
<td
bind:this={ref}
data-slot="table-cell"
class={cn(
"whitespace-nowrap bg-clip-padding p-2 align-middle [&:has([role=checkbox])]:pr-0",
className
)}
{...restProps}
>
{@render children?.()}
</td>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLTableSectionElement>> = $props();
</script>
<tfoot
bind:this={ref}
data-slot="table-footer"
class={cn("bg-muted/50 border-t font-medium [&>tr]:last:border-b-0", className)}
{...restProps}
>
{@render children?.()}
</tfoot>

View File

@@ -0,0 +1,23 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLThAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLThAttributes> = $props();
</script>
<th
bind:this={ref}
data-slot="table-head"
class={cn(
"text-foreground h-10 whitespace-nowrap bg-clip-padding px-2 text-left align-middle font-medium [&:has([role=checkbox])]:pr-0",
className
)}
{...restProps}
>
{@render children?.()}
</th>

View File

@@ -0,0 +1,20 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLTableSectionElement>> = $props();
</script>
<thead
bind:this={ref}
data-slot="table-header"
class={cn("[&_tr]:border-b", className)}
{...restProps}
>
{@render children?.()}
</thead>

View File

@@ -0,0 +1,23 @@
<script lang="ts">
import { cn, type WithElementRef } from "$lib/utils.js";
import type { HTMLAttributes } from "svelte/elements";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLAttributes<HTMLTableRowElement>> = $props();
</script>
<tr
bind:this={ref}
data-slot="table-row"
class={cn(
"hover:[&,&>svelte-css-wrapper]:[&>th,td]:bg-muted/50 data-[state=selected]:bg-muted border-b transition-colors",
className
)}
{...restProps}
>
{@render children?.()}
</tr>

View File

@@ -0,0 +1,22 @@
<script lang="ts">
import type { HTMLTableAttributes } from "svelte/elements";
import { cn, type WithElementRef } from "$lib/utils.js";
let {
ref = $bindable(null),
class: className,
children,
...restProps
}: WithElementRef<HTMLTableAttributes> = $props();
</script>
<div data-slot="table-container" class="relative w-full overflow-x-auto">
<table
bind:this={ref}
data-slot="table"
class={cn("w-full caption-bottom text-sm", className)}
{...restProps}
>
{@render children?.()}
</table>
</div>

View File

@@ -0,0 +1,143 @@
<script lang="ts">
import type { PageData } from './$types';
import * as Table from '$lib/components/ui/table';
import { Button } from '$lib/components/ui/button';
import * as DropdownMenu from '$lib/components/ui/dropdown-menu';
import { MoreHorizontal } from 'lucide-svelte';
import * as Dialog from '$lib/components/ui/dialog';
import IngestionSourceForm from '$lib/components/custom/IngestionSourceForm.svelte';
import { api } from '$lib/api';
import type { IngestionSource, CreateIngestionSourceDto } from '@open-archive/types';
let { data }: { data: PageData } = $props();
let ingestionSources = $state(data.ingestionSources);
let isDialogOpen = $state(false);
let selectedSource = $state<IngestionSource | null>(null);
const openCreateDialog = () => {
selectedSource = null;
isDialogOpen = true;
};
const openEditDialog = (source: IngestionSource) => {
selectedSource = source;
isDialogOpen = true;
};
const handleDelete = async (id: string) => {
if (!confirm('Are you sure you want to delete this ingestion source?')) {
return;
}
await api(`/ingestion-sources/${id}`, { method: 'DELETE' });
ingestionSources = ingestionSources.filter((s) => s.id !== id);
};
const handleSync = async (id: string) => {
await api(`/ingestion-sources/${id}/sync`, { method: 'POST' });
// Optionally, refresh the data or update the status locally
const updatedSources = ingestionSources.map((s) => {
if (s.id === id) {
return { ...s, status: 'syncing' as const };
}
return s;
});
ingestionSources = updatedSources;
};
const handleFormSubmit = async (formData: CreateIngestionSourceDto) => {
if (selectedSource) {
// Update
const response = await api(`/ingestion-sources/${selectedSource.id}`, {
method: 'PUT',
body: JSON.stringify(formData)
});
const updatedSource = await response.json();
ingestionSources = ingestionSources.map((s) =>
s.id === updatedSource.id ? updatedSource : s
);
} else {
// Create
const response = await api('/ingestion-sources', {
method: 'POST',
body: JSON.stringify(formData)
});
const newSource = await response.json();
ingestionSources = [...ingestionSources, newSource];
}
isDialogOpen = false;
};
</script>
<div class="container mx-auto py-10">
<div class="mb-4 flex items-center justify-between">
<h1 class="text-2xl font-bold">Ingestion Sources</h1>
<Button onclick={openCreateDialog}>Create New</Button>
</div>
<div class="rounded-md border">
<Table.Root>
<Table.Header>
<Table.Row>
<Table.Head>Name</Table.Head>
<Table.Head>Provider</Table.Head>
<Table.Head>Status</Table.Head>
<Table.Head>Created At</Table.Head>
<Table.Head class="text-right">Actions</Table.Head>
</Table.Row>
</Table.Header>
<Table.Body>
{#if ingestionSources.length > 0}
{#each ingestionSources as source (source.id)}
<Table.Row>
<Table.Cell>{source.name}</Table.Cell>
<Table.Cell>{source.provider}</Table.Cell>
<Table.Cell>{source.status}</Table.Cell>
<Table.Cell>{new Date(source.createdAt).toLocaleDateString()}</Table.Cell>
<Table.Cell class="text-right">
<DropdownMenu.Root>
<DropdownMenu.Trigger>
<Button variant="ghost" class="h-8 w-8 p-0">
<span class="sr-only">Open menu</span>
<MoreHorizontal class="h-4 w-4" />
</Button>
</DropdownMenu.Trigger>
<DropdownMenu.Content>
<DropdownMenu.Label>Actions</DropdownMenu.Label>
<DropdownMenu.Item onclick={() => openEditDialog(source)}
>Edit</DropdownMenu.Item
>
<DropdownMenu.Item onclick={() => handleSync(source.id)}>Sync</DropdownMenu.Item
>
<DropdownMenu.Separator />
<DropdownMenu.Item class="text-red-600" onclick={() => handleDelete(source.id)}
>Delete</DropdownMenu.Item
>
</DropdownMenu.Content>
</DropdownMenu.Root>
</Table.Cell>
</Table.Row>
{/each}
{:else}
<Table.Row>
<Table.Cell colspan={5} class="text-center">No ingestion sources found.</Table.Cell>
</Table.Row>
{/if}
</Table.Body>
</Table.Root>
</div>
</div>
<Dialog.Root bind:open={isDialogOpen}>
<Dialog.Content class="sm:max-w-[425px]">
<Dialog.Header>
<Dialog.Title>{selectedSource ? 'Edit' : 'Create'} Ingestion Source</Dialog.Title>
<Dialog.Description>
{selectedSource
? 'Make changes to your ingestion source here.'
: 'Add a new ingestion source to start archiving emails.'}
</Dialog.Description>
</Dialog.Header>
<IngestionSourceForm source={selectedSource} onSubmit={handleFormSubmit} />
</Dialog.Content>
</Dialog.Root>

View File

@@ -0,0 +1,22 @@
import { api } from '$lib/api';
import type { PageLoad } from './$types';
import type { IngestionSource } from '@open-archive/types';
export const load: PageLoad = async ({ fetch }) => {
try {
const response = await api('/ingestion-sources', {}, fetch);
if (!response.ok) {
throw new Error(`Failed to fetch ingestion sources: ${response.statusText}`);
}
const ingestionSources: IngestionSource[] = await response.json();
return {
ingestionSources
};
} catch (error) {
console.error('Failed to load ingestion sources:', error);
return {
ingestionSources: [],
error: 'Failed to load ingestion sources'
};
}
};

View File

@@ -1,2 +1,3 @@
export * from './auth.types';
export * from './user.types';
export * from './ingestion.types';

View File

@@ -0,0 +1,26 @@
export type IngestionProvider = 'google_workspace' | 'microsoft_365' | 'generic_imap';
export type IngestionStatus = 'active' | 'paused' | 'error' | 'pending_auth' | 'syncing';
export interface IngestionSource {
id: string;
name: string;
provider: IngestionProvider;
status: IngestionStatus;
createdAt: Date;
updatedAt: Date;
providerConfig: Record<string, any>;
}
export interface CreateIngestionSourceDto {
name: string;
provider: IngestionProvider;
providerConfig: Record<string, any>;
}
export interface UpdateIngestionSourceDto {
name?: string;
provider?: IngestionProvider;
status?: IngestionStatus;
providerConfig?: Record<string, any>;
}

File diff suppressed because one or more lines are too long