Add SMTP Journaling as an Ingestion Source for GoBD-Compliant Email Archiving #18

New Issue

MrUnknownDE · 2026-04-05T16:16:10+02:00

MrUnknownDE commented

2026-04-05 16:16:10 +02:00

Originally created by @gpappsoft on 3/7/2026

Summary

This proposal requests the addition of SMTP journaling as a dedicated ingestion method in OpenArchiver. SMTP journaling is the technically most reliable and legally most defensible approach to capturing all email traffic in real time — a requirement under GoBD (Grundsätze zur ordnungsmäßigen Führung und Aufbewahrung von Büchern, Aufzeichnungen und Unterlagen in elektronischer Form sowie zum Datenzugriff), the German principles governing audit-proof electronic record keeping.

Background & Motivation

OpenArchiver currently supports ingestion via IMAP sync, Microsoft Graph API, and file uploads (PST/EML). While these methods are useful for bulk imports and continuous mailbox synchronization, they have a fundamental limitation from a compliance perspective: they operate at the mailbox level after delivery, which means emails can theoretically be deleted, altered, or missed before the archiver captures them.

Under GoBD, businesses in Germany are required to archive all tax-relevant business correspondence in a manner that is:

Complete — no email may be missed
Immutable — emails must be stored in their original, unaltered form
Timely — capture must happen as close to transmission as possible
Traceable — all access and changes must be logged

Journaling is the most reliable and legally secure method for this, as the email server automatically creates an additional copy of every incoming and outgoing email during transport — before the message reaches or leaves the user's mailbox. This means even if a user deletes an email immediately after receiving it, the journal copy has already been safely captured.

IMAP-based retrieval, by contrast, is suitable for retrieving historical data but cannot guarantee complete real-time capture. Only journaling can provide the completeness guarantee required for full GoBD compliance.

Proposed Solution

Implement an SMTP journaling receiver as a new ingestion connector in OpenArchiver. The system would work as follows:

1. SMTP Listener Service

OpenArchiver exposes a dedicated SMTP endpoint (e.g., on port 25 or a configurable port). Mail servers (Exchange, Postfix, Zimbra, Google Workspace, Microsoft 365, etc.) are configured to BCC or forward a journal copy of every inbound/outbound message to this endpoint.

2. Ingestion Pipeline

Messages arriving at the SMTP listener are immediately:

Parsed and stored as .eml files (preserving the original format, as required by GoBD)
SHA-256 hashed to generate a cryptographic fingerprint for integrity verification
Timestamped server-side (independent of the sending MTA's clock)
Deduplicated against existing archives using the existing deduplication logic

3. TLS Encryption in Transit

The SMTP listener must enforce TLS to protect email contents during the journaling relay, preventing tampering in transit.

4. Audit Logging

Every ingestion event via the journal endpoint is written to an append-only audit log, recording source IP, message ID, timestamp, and hash — supporting the traceability requirement of GoBD.

GoBD Compliance Checklist Addressed by This Feature

Requirement	How SMTP Journaling Addresses It
Completeness	Capture happens at transport layer, before any user action
Immutability	Emails stored as `.eml` + SHA-256 hash; no post-ingest modification
Original format	Raw MIME message preserved, not re-encoded
Timeliness	Real-time capture during SMTP relay
Traceability	Append-only ingestion audit log
Retention enforcement	Existing retention policy engine applies

Compatibility

SMTP journaling is supported natively by all major mail server platforms:

Mail Server	Configuration Method
Microsoft Exchange / Microsoft 365	Journal rule targeting the OpenArchiver SMTP endpoint
Google Workspace	Vault-style routing or SMTP relay forwarding
Postfix	`always_bcc` or `recipient_bcc_maps` directive
Zimbra	Built-in archiving/journaling via zimlet or MTA config
Dovecot + Postfix	Custom transport rule

This makes the feature mail-server-agnostic and valuable for a wide range of self-hosted deployments.

Deliverables / Scope

SMTP listener service (configurable port, optional TLS, optional SMTP AUTH)
Journal ingestion pipeline (parse → hash → deduplicate → store)
UI: new "SMTP Journal" connector type in the ingestion settings
Audit log viewer for journal ingestion events
Documentation: setup guides for Exchange, Microsoft 365, Postfix, Google Workspace
GoBD compliance documentation describing how the feature satisfies each requirement

Why This Matters for OpenArchiver

OpenArchiver's stated goal is legally compliant email archiving. For German businesses (and businesses operating under comparable regulations in Austria and Switzerland), SMTP journaling is not a "nice to have" — it is the only method that fully satisfies GoBD. Adding this feature would:

Make OpenArchiver a complete, production-ready GoBD compliance solution
Differentiate it from IMAP-only archiving tools
Enable self-hosted deployment as a full alternative to commercial solutions like MailStore Server, REDDOXX MailDepot, or Hornetsecurity Email Archiving

💡 A Note on Market Opportunity for Open-Source Businesses

For open-source companies or freelancers offering managed hosting, support, or consulting services around OpenArchiver — particularly those based in Germany, Austria, or Switzerland — this feature represents a significant commercial opportunity.

GoBD compliance is not optional for German businesses. Every company subject to German tax law that uses email for business correspondence is legally required to archive it in an audit-proof manner. This affects hundreds of thousands of SMEs across Germany alone. The market for GoBD-compliant email archiving is well-established, currently dominated by proprietary commercial solutions that often come with significant licensing costs.

A self-hosted, open-source archiving solution with verifiable SMTP journaling and GoBD compliance would be a compelling offering for:

IT service providers and MSPs looking to offer a cost-effective, auditable compliance solution to their SME clients
System houses (Systemhäuser) that prefer open-source stacks for vendor independence and transparency
Open-source consultancies selling installation, configuration, and support contracts around OpenArchiver
Hosting providers offering OpenArchiver as a managed service in German data centers — a meaningful trust signal for data-sensitive clients

The key selling point is straightforward:

"Self-hosted, fully open-source, GoBD-compliant email archiving — no vendor lock-in, no per-user licensing, full auditability of the software itself."

This message resonates strongly with German IT buyers, who place high value on data sovereignty, transparency, and regulatory certainty.

Adding SMTP journaling would be the final piece that transforms OpenArchiver from a useful tool into a complete GoBD compliance solution — and a serious open-source alternative to commercial products in a market where such alternatives are currently almost nonexistent.

*Originally created by @gpappsoft on 3/7/2026* ## Summary This proposal requests the addition of **SMTP journaling** as a dedicated ingestion method in OpenArchiver. SMTP journaling is the technically most reliable and legally most defensible approach to capturing all email traffic in real time — a requirement under **GoBD** (Grundsätze zur ordnungsmäßigen Führung und Aufbewahrung von Büchern, Aufzeichnungen und Unterlagen in elektronischer Form sowie zum Datenzugriff), the German principles governing audit-proof electronic record keeping. --- ## Background & Motivation OpenArchiver currently supports ingestion via IMAP sync, Microsoft Graph API, and file uploads (PST/EML). While these methods are useful for bulk imports and continuous mailbox synchronization, they have a fundamental limitation from a compliance perspective: **they operate at the mailbox level after delivery**, which means emails can theoretically be deleted, altered, or missed before the archiver captures them. Under GoBD, businesses in Germany are required to archive all tax-relevant business correspondence in a manner that is: - **Complete** — no email may be missed - **Immutable** — emails must be stored in their original, unaltered form - **Timely** — capture must happen as close to transmission as possible - **Traceable** — all access and changes must be logged Journaling is the most reliable and legally secure method for this, as the email server automatically creates an additional copy of every incoming and outgoing email **during transport** — before the message reaches or leaves the user's mailbox. This means even if a user deletes an email immediately after receiving it, the journal copy has already been safely captured. IMAP-based retrieval, by contrast, is suitable for retrieving historical data but **cannot guarantee complete real-time capture**. Only journaling can provide the completeness guarantee required for full GoBD compliance. --- ## Proposed Solution Implement an **SMTP journaling receiver** as a new ingestion connector in OpenArchiver. The system would work as follows: ### 1. SMTP Listener Service OpenArchiver exposes a dedicated SMTP endpoint (e.g., on port 25 or a configurable port). Mail servers (Exchange, Postfix, Zimbra, Google Workspace, Microsoft 365, etc.) are configured to BCC or forward a journal copy of every inbound/outbound message to this endpoint. ### 2. Ingestion Pipeline Messages arriving at the SMTP listener are immediately: - Parsed and stored as `.eml` files (preserving the original format, as required by GoBD) - SHA-256 hashed to generate a cryptographic fingerprint for integrity verification - Timestamped server-side (independent of the sending MTA's clock) - Deduplicated against existing archives using the existing deduplication logic ### 3. TLS Encryption in Transit The SMTP listener must enforce TLS to protect email contents during the journaling relay, preventing tampering in transit. ### 4. Audit Logging Every ingestion event via the journal endpoint is written to an append-only audit log, recording source IP, message ID, timestamp, and hash — supporting the traceability requirement of GoBD. --- ## GoBD Compliance Checklist Addressed by This Feature | Requirement | How SMTP Journaling Addresses It | |---|---| | Completeness | Capture happens at transport layer, before any user action | | Immutability | Emails stored as `.eml` + SHA-256 hash; no post-ingest modification | | Original format | Raw MIME message preserved, not re-encoded | | Timeliness | Real-time capture during SMTP relay | | Traceability | Append-only ingestion audit log | | Retention enforcement | Existing retention policy engine applies | --- ## Compatibility SMTP journaling is supported natively by all major mail server platforms: | Mail Server | Configuration Method | |---|---| | **Microsoft Exchange / Microsoft 365** | Journal rule targeting the OpenArchiver SMTP endpoint | | **Google Workspace** | Vault-style routing or SMTP relay forwarding | | **Postfix** | `always_bcc` or `recipient_bcc_maps` directive | | **Zimbra** | Built-in archiving/journaling via zimlet or MTA config | | **Dovecot + Postfix** | Custom transport rule | This makes the feature **mail-server-agnostic** and valuable for a wide range of self-hosted deployments. --- ## Deliverables / Scope - [ ] SMTP listener service (configurable port, optional TLS, optional SMTP AUTH) - [ ] Journal ingestion pipeline (parse → hash → deduplicate → store) - [ ] UI: new "SMTP Journal" connector type in the ingestion settings - [ ] Audit log viewer for journal ingestion events - [ ] Documentation: setup guides for Exchange, Microsoft 365, Postfix, Google Workspace - [ ] GoBD compliance documentation describing how the feature satisfies each requirement --- ## Why This Matters for OpenArchiver OpenArchiver's stated goal is *legally compliant email archiving*. For German businesses (and businesses operating under comparable regulations in Austria and Switzerland), SMTP journaling is not a "nice to have" — it is the **only method that fully satisfies GoBD**. Adding this feature would: 1. Make OpenArchiver a complete, production-ready GoBD compliance solution 2. Differentiate it from IMAP-only archiving tools 3. Enable self-hosted deployment as a full alternative to commercial solutions like MailStore Server, REDDOXX MailDepot, or Hornetsecurity Email Archiving --- ## 💡 A Note on Market Opportunity for Open-Source Businesses For open-source companies or freelancers offering managed hosting, support, or consulting services around OpenArchiver — particularly those based in **Germany, Austria, or Switzerland** — this feature represents a significant commercial opportunity. GoBD compliance is not optional for German businesses. Every company subject to German tax law that uses email for business correspondence is **legally required** to archive it in an audit-proof manner. This affects hundreds of thousands of SMEs across Germany alone. The market for GoBD-compliant email archiving is well-established, currently dominated by proprietary commercial solutions that often come with significant licensing costs. A self-hosted, open-source archiving solution with verifiable SMTP journaling and GoBD compliance would be a compelling offering for: - **IT service providers and MSPs** looking to offer a cost-effective, auditable compliance solution to their SME clients - **System houses** (*Systemhäuser*) that prefer open-source stacks for vendor independence and transparency - **Open-source consultancies** selling installation, configuration, and support contracts around OpenArchiver - **Hosting providers** offering OpenArchiver as a managed service in German data centers — a meaningful trust signal for data-sensitive clients The key selling point is straightforward: > *"Self-hosted, fully open-source, GoBD-compliant email archiving — no vendor lock-in, no per-user licensing, full auditability of the software itself."* This message resonates strongly with German IT buyers, who place high value on **data sovereignty, transparency, and regulatory certainty**. Adding SMTP journaling would be the final piece that transforms OpenArchiver from a useful tool into a **complete GoBD compliance solution** — and a serious open-source alternative to commercial products in a market where such alternatives are currently almost nonexistent. ---

MrUnknownDE added the enhancement enhancement enhancement labels 2026-04-05 16:16:10 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/OpenArchiver#18