IMAP import job hangs due to large starting UID and inefficient UID iteration #151

New Issue

MrUnknownDE · 2026-04-05T16:16:45+02:00

MrUnknownDE commented

2026-04-05 16:16:45 +02:00

Originally created by @prinsss on 10/15/2025

Describe the bug

When the IMAP server uses very large UIDs (e.g. { seq: 1, uid: 1377008892 }), the import job in ImapConnector iterates from UID 0 up to the actual lowest valid UID. This causes many unnecessary fetch commands, significantly slowing down the import process or causing it to hang indefinitely.

According to the IMAP standard, UIDs do not always start from 0 or 1; servers may assign any positive integer as the lowest UID. The current logic assumes a starting UID of 0, which is inefficient for such cases.

1e048fdbc1/packages/backend/src/services/ingestion-connectors/ImapConnector.ts (L207)

To Reproduce

Connect to an IMAP server where mailbox messages have very large UIDs (e.g., starting at 1377008892).
Run the import job with ImapConnector.
Observe the import job sending many fetch commands for UIDs that do not exist, causing delays or hanging.

Expected behavior

The import job should detect the actual lowest UID available in the mailbox and start iteration from there, rather than assuming UID 0.

System:

Open Archiver Version: 0.3.4

Relevant logs:

if (mailbox.exists > 0) {
  const lastMessage = await this.client.fetchOne(String(mailbox.exists), {
    uid: true,
  });
+ console.log('lastMessage', lastMessage);
  if (lastMessage && lastMessage.uid > currentMaxUid) {
    currentMaxUid = lastMessage.uid;
+   logger.info('Updating currentMaxUid to %d for mailbox %s', currentMaxUid, mailboxPath);
  }
+
+ const firstMessage = await this.client.fetchOne('1', { uid: true });
+ console.log('firstMessage', firstMessage);
}

👉 Debug logs:

[0] lastMessage { seq: 326, uid: 1377009217, id: '63b60e410a1db7d600852a7e6f27e208' }
[0] [15:15:50.718] DEBUG (47981): 9 OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.718] INFO (47981): Updating currentMaxUid to 1377009217 for mailbox INBOX
[0] [15:15:50.718] DEBUG (47981): A FETCH 1 UID
[0]     module: "ImapFlow"
[0]     src: "s"
[0] firstMessage { seq: 1, uid: 1377008892, id: '4e3f698708ca14de5e28cc2a7a307ab5' }
[0] [15:15:50.753] DEBUG (47981): A OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.753] INFO (47981): startUid <= maxUidToFetch: 1 <= 1377009217
[0] [15:15:50.754] DEBUG (47981): B UID FETCH 1:250 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[])
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.787] DEBUG (47981): B OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.788] INFO (47981): startUid <= maxUidToFetch: 251 <= 1377009217
[0] [15:15:50.788] DEBUG (47981): C UID FETCH 251:500 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[])
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.823] DEBUG (47981): C OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.824] INFO (47981): startUid <= maxUidToFetch: 501 <= 1377009217
[0] [15:15:50.824] DEBUG (47981): D UID FETCH 501:750 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[])
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.856] DEBUG (47981): D OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"

[... 5000+ lines of logs ...]

[0] [15:16:10.178] INFO (47981): startUid <= maxUidToFetch: 136501 <= 1377009217
[0] [15:16:10.178] DEBUG (47981): 22D UID FETCH 136501:136750 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[])
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:16:10.217] DEBUG (47981): 22D OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"

Additional context

AFAIK, imap.163.com is using very large UIDs.

*Originally created by @prinsss on 10/15/2025* **Describe the bug** When the IMAP server uses very large UIDs (e.g. `{ seq: 1, uid: 1377008892 }`), the import job in ImapConnector iterates from UID 0 up to the actual lowest valid UID. This causes many unnecessary fetch commands, significantly slowing down the import process or causing it to hang indefinitely. According to the [IMAP standard](https://datatracker.ietf.org/doc/html/rfc3501#section-2.3.1.1), UIDs do not always start from 0 or 1; servers may assign any positive integer as the lowest UID. The current logic assumes a starting UID of 0, which is inefficient for such cases. https://github.com/LogicLabs-OU/OpenArchiver/blob/1e048fdbc1d0e60b35a6c633309acd14927a6824/packages/backend/src/services/ingestion-connectors/ImapConnector.ts#L207 **To Reproduce** 1. Connect to an IMAP server where mailbox messages have very large UIDs (e.g., starting at 1377008892). 2. Run the import job with ImapConnector. 3. Observe the import job sending many fetch commands for UIDs that do not exist, causing delays or hanging. **Expected behavior** The import job should detect the actual lowest UID available in the mailbox and start iteration from there, rather than assuming UID 0. **System:** - Open Archiver Version: 0.3.4 **Relevant logs:** ```diff if (mailbox.exists > 0) { const lastMessage = await this.client.fetchOne(String(mailbox.exists), { uid: true, }); + console.log('lastMessage', lastMessage); if (lastMessage && lastMessage.uid > currentMaxUid) { currentMaxUid = lastMessage.uid; + logger.info('Updating currentMaxUid to %d for mailbox %s', currentMaxUid, mailboxPath); } + + const firstMessage = await this.client.fetchOne('1', { uid: true }); + console.log('firstMessage', firstMessage); } ``` <details> <summary>👉 Debug logs:</summary> ``` [0] lastMessage { seq: 326, uid: 1377009217, id: '63b60e410a1db7d600852a7e6f27e208' } [0] [15:15:50.718] DEBUG (47981): 9 OK Fetch completed [0] module: "ImapFlow" [0] src: "s" [0] [15:15:50.718] INFO (47981): Updating currentMaxUid to 1377009217 for mailbox INBOX [0] [15:15:50.718] DEBUG (47981): A FETCH 1 UID [0] module: "ImapFlow" [0] src: "s" [0] firstMessage { seq: 1, uid: 1377008892, id: '4e3f698708ca14de5e28cc2a7a307ab5' } [0] [15:15:50.753] DEBUG (47981): A OK Fetch completed [0] module: "ImapFlow" [0] src: "s" [0] [15:15:50.753] INFO (47981): startUid <= maxUidToFetch: 1 <= 1377009217 [0] [15:15:50.754] DEBUG (47981): B UID FETCH 1:250 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[]) [0] module: "ImapFlow" [0] src: "s" [0] [15:15:50.787] DEBUG (47981): B OK Fetch completed [0] module: "ImapFlow" [0] src: "s" [0] [15:15:50.788] INFO (47981): startUid <= maxUidToFetch: 251 <= 1377009217 [0] [15:15:50.788] DEBUG (47981): C UID FETCH 251:500 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[]) [0] module: "ImapFlow" [0] src: "s" [0] [15:15:50.823] DEBUG (47981): C OK Fetch completed [0] module: "ImapFlow" [0] src: "s" [0] [15:15:50.824] INFO (47981): startUid <= maxUidToFetch: 501 <= 1377009217 [0] [15:15:50.824] DEBUG (47981): D UID FETCH 501:750 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[]) [0] module: "ImapFlow" [0] src: "s" [0] [15:15:50.856] DEBUG (47981): D OK Fetch completed [0] module: "ImapFlow" [0] src: "s" [... 5000+ lines of logs ...] [0] [15:16:10.178] INFO (47981): startUid <= maxUidToFetch: 136501 <= 1377009217 [0] [15:16:10.178] DEBUG (47981): 22D UID FETCH 136501:136750 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[]) [0] module: "ImapFlow" [0] src: "s" [0] [15:16:10.217] DEBUG (47981): 22D OK Fetch completed [0] module: "ImapFlow" [0] src: "s" ``` </details> **Additional context** AFAIK, `imap.163.com` is using very large UIDs.

MrUnknownDE added the bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug labels 2026-04-05 16:16:46 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/OpenArchiver#151