SIGTERM and container exitting at least once a day #1172

New Issue

MrUnknownDE · 2026-04-06T01:41:22+02:00

MrUnknownDE commented

2026-04-06 01:41:22 +02:00

Originally created by @ghanjiboy on 4/5/2024

Subject of the issue

Over the last 2-3 months, I've noticed my vaultwarden container getting a SIGTERM and exiting. I am not terminating it and I don't see any errors or access patterns that would cause it to exit. I have enabled tracing logs to see if it would show something more, but not seeing anything indicative of an issue. The logs are mostly similar to the following excerpt:

[2024-04-05 05:01:38.445][tracing::span][TRACE] parse_headers;
[2024-04-05 05:01:38.445][tracing::span::active][TRACE] -> parse_headers;
[2024-04-05 05:01:38.445][tracing::span::active][TRACE] <- parse_headers;
[2024-04-05 05:01:38.446][tracing::span][TRACE] -- parse_headers;
[2024-04-05 05:01:38.446][request][INFO] GET /alive
[2024-04-05 05:01:38.449][response][INFO] (alive) GET /alive => 200 OK
[2024-04-05 05:01:38.450][tracing::span][TRACE] encode_headers;
[2024-04-05 05:01:38.450][tracing::span::active][TRACE] -> encode_headers;
[2024-04-05 05:01:38.450][tracing::span::active][TRACE] <- encode_headers;
[2024-04-05 05:01:38.450][tracing::span][TRACE] -- encode_headers;
[2024-04-05 05:01:40.027][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:01:40.028][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:02:38.689][tracing::span][TRACE] parse_headers;
[2024-04-05 05:02:38.690][tracing::span::active][TRACE] -> parse_headers;
[2024-04-05 05:02:38.690][tracing::span::active][TRACE] <- parse_headers;
[2024-04-05 05:02:38.690][tracing::span][TRACE] -- parse_headers;
[2024-04-05 05:02:38.690][request][INFO] GET /alive
[2024-04-05 05:02:38.692][response][INFO] (alive) GET /alive => 200 OK
[2024-04-05 05:02:38.692][tracing::span][TRACE] encode_headers;
[2024-04-05 05:02:38.693][tracing::span::active][TRACE] -> encode_headers;
[2024-04-05 05:02:38.693][tracing::span::active][TRACE] <- encode_headers;
[2024-04-05 05:02:38.693][tracing::span][TRACE] -- encode_headers;
[2024-04-05 05:02:40.031][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:02:40.031][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:03:10.034][vaultwarden::api::core::emergency_access][DEBUG] Start emergency_notification_reminder_job
[2024-04-05 05:03:10.041][vaultwarden::api::core::emergency_access][DEBUG] No emergency request reminder notification to send
[2024-04-05 05:03:40.035][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:03:40.035][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:04:40.037][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:04:40.038][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:05:10.040][vaultwarden::api::core::sends][DEBUG] Purging sends
[2024-04-05 05:05:40.042][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:05:40.043][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:06:11.431][rocket::server][WARN] Received SIGTERM. Requesting shutdown.
[2024-04-05 05:06:11.432][vaultwarden][INFO] Vaultwarden process exited!

Deployment environment

Your environment (Generated via diagnostics page)

Vaultwarden version: v1.30.5
Web-vault version: v2024.1.2b
OS/Arch: linux/x86_64
Running within a container: true (Base: Debian)
Environment settings overridden: true
Uses a reverse proxy: true
IP Header check: false (X-Forwarded-For)
Internet access: true
Internet access via a proxy: false
DNS Check: true
Browser/Server Time Check: true
Server/NTP Time Check: true
Domain Configuration Check: true
HTTPS Check: true
Database type: PostgreSQL
Database version: PostgreSQL 14.0 (Debian 14.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
Clients used:
Reverse proxy and version:
Other relevant information:

Config (Generated via diagnostics page)

Show Running Config

Environment settings which are overridden: DOMAIN, ADMIN_TOKEN, SMTP_HOST, SMTP_PORT, SMTP_FROM

{
  "_duo_akey": null,
  "_enable_duo": false,
  "_enable_email_2fa": true,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_icon_service_csp": "",
  "_icon_service_url": "",
  "_ip_header_enabled": true,
  "_smtp_img_src": "cid:",
  "admin_ratelimit_max_burst": 3,
  "admin_ratelimit_seconds": 300,
  "admin_session_lifetime": 20,
  "admin_token": "***",
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "auth_request_purge_schedule": "30 * * * * *",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_conn_init": "",
  "database_max_conns": 10,
  "database_timeout": 30,
  "database_url": "**********://*****************************************************",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": false,
  "disable_icon_download": false,
  "domain": "*****://****************",
  "domain_origin": "*****://****************",
  "domain_path": "",
  "domain_set": true,
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "email_attempts_limit": 3,
  "email_change_allowed": true,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "emergency_access_allowed": true,
  "emergency_notification_reminder_schedule": "0 3 * * * *",
  "emergency_request_timeout_schedule": "0 7 * * * *",
  "enable_db_wal": false,
  "event_cleanup_schedule": "0 10 0 * * *",
  "events_days_retain": null,
  "experimental_client_feature_flags": "fido2-vault-credentials",
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": "***",
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "data/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "icon_redirect_code": 302,
  "icon_service": "internal",
  "incomplete_2fa_schedule": "30 * * * * *",
  "incomplete_2fa_time_limit": 3,
  "invitation_expiration_hours": 120,
  "invitation_org_name": "Bitwarden_RS",
  "invitations_allowed": false,
  "ip_header": "X-Real-IP",
  "job_poll_interval_ms": 30000,
  "log_file": null,
  "log_level": "trace",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "login_ratelimit_max_burst": 10,
  "login_ratelimit_seconds": 60,
  "org_attachment_limit": null,
  "org_creation_users": "",
  "org_events_enabled": false,
  "org_groups_enabled": false,
  "password_hints_allowed": true,
  "password_iterations": 100000,
  "push_enabled": false,
  "push_identity_uri": "https://identity.bitwarden.com",
  "push_installation_id": "***",
  "push_installation_key": "***",
  "push_relay_uri": "https://push.bitwarden.com",
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "data/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sendmail_command": null,
  "sends_allowed": true,
  "sends_folder": "data/sends",
  "show_password_hint": false,
  "signups_allowed": true,
  "signups_domains_whitelist": "",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_embed_images": true,
  "smtp_explicit_tls": null,
  "smtp_from": "*********************",
  "smtp_from_name": "Bitwarden_RS",
  "smtp_host": "************",
  "smtp_password": null,
  "smtp_port": 25,
  "smtp_security": "off",
  "smtp_ssl": null,
  "smtp_timeout": 15,
  "smtp_username": null,
  "templates_folder": "data/templates",
  "tmp_folder": "data/tmp",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_sendmail": false,
  "use_syslog": false,
  "user_attachment_limit": null,
  "user_send_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "websocket_address": "0.0.0.0",
  "websocket_enabled": false,
  "websocket_port": 3012,
  "yubico_client_id": null,
  "yubico_secret_key": null,
  "yubico_server": null
}

vaultwarden version:

Install method:
I've been using vaultwarden in a docker swarm deployment for at least 3 years - I pull the latest probably shortly after the release is made.
Clients used:
I use web client, IOS BW app, Android BW app - all on latest as well.
Reverse proxy and version:
nginx
MySQL/MariaDB or PostgreSQL version:
PostgreSQL 14.0 (Debian 14.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
Other relevant details:

Steps to reproduce

Docker swarm takes care of ensuring there is 1 container of vaultwarden running
Every day, at different times, at least once, a SIGTERM will be logged and the docker container will exit
A new container will spin up shortly after that

Expected behaviour

The container should stay up and running

Actual behaviour

The container abruptly stops.

Troubleshooting data

I am able to look back at logs for up to last 30 days and this is the pattern of SIGTERM occurrences:

04/05/2024 2
04/04/2024 2
04/03/2024 1
04/02/2024 3
04/01/2024 1
03/31/2024 4
03/30/2024 3
03/29/2024 1
03/28/2024 3
03/27/2024 5
03/26/2024 3
03/25/2024 5
03/24/2024 5
03/23/2024 4
03/22/2024 3
03/21/2024 2
03/20/2024 3
03/19/2024 3
03/18/2024 5
03/17/2024 4
03/16/2024 3
03/15/2024 4
03/14/2024 3
03/13/2024 2
03/12/2024 3
03/11/2024 3
03/10/2024 1
03/09/2024 3
03/08/2024 2
03/07/2024 3

*Originally created by @ghanjiboy on 4/5/2024*   ### Subject of the issue Over the last 2-3 months, I've noticed my vaultwarden container getting a SIGTERM and exiting. I am not terminating it and I don't see any errors or access patterns that would cause it to exit. I have enabled tracing logs to see if it would show something more, but not seeing anything indicative of an issue. The logs are mostly similar to the following excerpt: [2024-04-05 05:01:38.445][tracing::span][TRACE] parse_headers; [2024-04-05 05:01:38.445][tracing::span::active][TRACE] -> parse_headers; [2024-04-05 05:01:38.445][tracing::span::active][TRACE] <- parse_headers; [2024-04-05 05:01:38.446][tracing::span][TRACE] -- parse_headers; [2024-04-05 05:01:38.446][request][INFO] GET /alive [2024-04-05 05:01:38.449][response][INFO] (alive) GET /alive => 200 OK [2024-04-05 05:01:38.450][tracing::span][TRACE] encode_headers; [2024-04-05 05:01:38.450][tracing::span::active][TRACE] -> encode_headers; [2024-04-05 05:01:38.450][tracing::span::active][TRACE] <- encode_headers; [2024-04-05 05:01:38.450][tracing::span][TRACE] -- encode_headers; [2024-04-05 05:01:40.027][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins [2024-04-05 05:01:40.028][vaultwarden::api::core::accounts][DEBUG] Purging auth requests [2024-04-05 05:02:38.689][tracing::span][TRACE] parse_headers; [2024-04-05 05:02:38.690][tracing::span::active][TRACE] -> parse_headers; [2024-04-05 05:02:38.690][tracing::span::active][TRACE] <- parse_headers; [2024-04-05 05:02:38.690][tracing::span][TRACE] -- parse_headers; [2024-04-05 05:02:38.690][request][INFO] GET /alive [2024-04-05 05:02:38.692][response][INFO] (alive) GET /alive => 200 OK [2024-04-05 05:02:38.692][tracing::span][TRACE] encode_headers; [2024-04-05 05:02:38.693][tracing::span::active][TRACE] -> encode_headers; [2024-04-05 05:02:38.693][tracing::span::active][TRACE] <- encode_headers; [2024-04-05 05:02:38.693][tracing::span][TRACE] -- encode_headers; [2024-04-05 05:02:40.031][vaultwarden::api::core::accounts][DEBUG] Purging auth requests [2024-04-05 05:02:40.031][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins [2024-04-05 05:03:10.034][vaultwarden::api::core::emergency_access][DEBUG] Start emergency_notification_reminder_job [2024-04-05 05:03:10.041][vaultwarden::api::core::emergency_access][DEBUG] No emergency request reminder notification to send [2024-04-05 05:03:40.035][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins [2024-04-05 05:03:40.035][vaultwarden::api::core::accounts][DEBUG] Purging auth requests [2024-04-05 05:04:40.037][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins [2024-04-05 05:04:40.038][vaultwarden::api::core::accounts][DEBUG] Purging auth requests [2024-04-05 05:05:10.040][vaultwarden::api::core::sends][DEBUG] Purging sends [2024-04-05 05:05:40.042][vaultwarden::api::core::accounts][DEBUG] Purging auth requests [2024-04-05 05:05:40.043][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins [2024-04-05 05:06:11.431][rocket::server][WARN] Received SIGTERM. Requesting shutdown. [2024-04-05 05:06:11.432][vaultwarden][INFO] Vaultwarden process exited! ### Deployment environment ### Your environment (Generated via diagnostics page) * Vaultwarden version: v1.30.5 * Web-vault version: v2024.1.2b * OS/Arch: linux/x86_64 * Running within a container: true (Base: Debian) * Environment settings overridden: true * Uses a reverse proxy: true * IP Header check: false (X-Forwarded-For) * Internet access: true * Internet access via a proxy: false * DNS Check: true * Browser/Server Time Check: true * Server/NTP Time Check: true * Domain Configuration Check: true * HTTPS Check: true * Database type: PostgreSQL * Database version: PostgreSQL 14.0 (Debian 14.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit * Clients used: * Reverse proxy and version: * Other relevant information: ### Config (Generated via diagnostics page) <details><summary>Show Running Config</summary> **Environment settings which are overridden:** DOMAIN, ADMIN_TOKEN, SMTP_HOST, SMTP_PORT, SMTP_FROM ```json { "_duo_akey": null, "_enable_duo": false, "_enable_email_2fa": true, "_enable_smtp": true, "_enable_yubico": true, "_icon_service_csp": "", "_icon_service_url": "", "_ip_header_enabled": true, "_smtp_img_src": "cid:", "admin_ratelimit_max_burst": 3, "admin_ratelimit_seconds": 300, "admin_session_lifetime": 20, "admin_token": "***", "allowed_iframe_ancestors": "", "attachments_folder": "data/attachments", "auth_request_purge_schedule": "30 * * * * *", "authenticator_disable_time_drift": false, "data_folder": "data", "database_conn_init": "", "database_max_conns": 10, "database_timeout": 30, "database_url": "**********://*****************************************************", "db_connection_retries": 15, "disable_2fa_remember": false, "disable_admin_token": false, "disable_icon_download": false, "domain": "*****://****************", "domain_origin": "*****://****************", "domain_path": "", "domain_set": true, "duo_host": null, "duo_ikey": null, "duo_skey": null, "email_attempts_limit": 3, "email_change_allowed": true, "email_expiration_time": 600, "email_token_size": 6, "emergency_access_allowed": true, "emergency_notification_reminder_schedule": "0 3 * * * *", "emergency_request_timeout_schedule": "0 7 * * * *", "enable_db_wal": false, "event_cleanup_schedule": "0 10 0 * * *", "events_days_retain": null, "experimental_client_feature_flags": "fido2-vault-credentials", "extended_logging": true, "helo_name": null, "hibp_api_key": "***", "icon_blacklist_non_global_ips": true, "icon_blacklist_regex": null, "icon_cache_folder": "data/icon_cache", "icon_cache_negttl": 259200, "icon_cache_ttl": 2592000, "icon_download_timeout": 10, "icon_redirect_code": 302, "icon_service": "internal", "incomplete_2fa_schedule": "30 * * * * *", "incomplete_2fa_time_limit": 3, "invitation_expiration_hours": 120, "invitation_org_name": "Bitwarden_RS", "invitations_allowed": false, "ip_header": "X-Real-IP", "job_poll_interval_ms": 30000, "log_file": null, "log_level": "trace", "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f", "login_ratelimit_max_burst": 10, "login_ratelimit_seconds": 60, "org_attachment_limit": null, "org_creation_users": "", "org_events_enabled": false, "org_groups_enabled": false, "password_hints_allowed": true, "password_iterations": 100000, "push_enabled": false, "push_identity_uri": "https://identity.bitwarden.com", "push_installation_id": "***", "push_installation_key": "***", "push_relay_uri": "https://push.bitwarden.com", "reload_templates": false, "require_device_email": false, "rsa_key_filename": "data/rsa_key", "send_purge_schedule": "0 5 * * * *", "sendmail_command": null, "sends_allowed": true, "sends_folder": "data/sends", "show_password_hint": false, "signups_allowed": true, "signups_domains_whitelist": "", "signups_verify": false, "signups_verify_resend_limit": 6, "signups_verify_resend_time": 3600, "smtp_accept_invalid_certs": false, "smtp_accept_invalid_hostnames": false, "smtp_auth_mechanism": null, "smtp_debug": false, "smtp_embed_images": true, "smtp_explicit_tls": null, "smtp_from": "*********************", "smtp_from_name": "Bitwarden_RS", "smtp_host": "************", "smtp_password": null, "smtp_port": 25, "smtp_security": "off", "smtp_ssl": null, "smtp_timeout": 15, "smtp_username": null, "templates_folder": "data/templates", "tmp_folder": "data/tmp", "trash_auto_delete_days": null, "trash_purge_schedule": "0 5 0 * * *", "use_sendmail": false, "use_syslog": false, "user_attachment_limit": null, "user_send_limit": null, "web_vault_enabled": true, "web_vault_folder": "web-vault/", "websocket_address": "0.0.0.0", "websocket_enabled": false, "websocket_port": 3012, "yubico_client_id": null, "yubico_secret_key": null, "yubico_server": null } ``` </details>     * vaultwarden version:  * Install method: * I've been using vaultwarden in a docker swarm deployment for at least 3 years - I pull the latest probably shortly after the release is made. * Clients used:  I use web client, IOS BW app, Android BW app - all on latest as well. * Reverse proxy and version:  nginx * MySQL/MariaDB or PostgreSQL version:  PostgreSQL 14.0 (Debian 14.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit * Other relevant details: ### Steps to reproduce  1. Docker swarm takes care of ensuring there is 1 container of vaultwarden running 2. Every day, at different times, at least once, a SIGTERM will be logged and the docker container will exit 3. A new container will spin up shortly after that ### Expected behaviour  The container should stay up and running ### Actual behaviour  The container abruptly stops. ### Troubleshooting data  I am able to look back at logs for up to last 30 days and this is the pattern of SIGTERM occurrences: ``` 04/05/2024 2 04/04/2024 2 04/03/2024 1 04/02/2024 3 04/01/2024 1 03/31/2024 4 03/30/2024 3 03/29/2024 1 03/28/2024 3 03/27/2024 5 03/26/2024 3 03/25/2024 5 03/24/2024 5 03/23/2024 4 03/22/2024 3 03/21/2024 2 03/20/2024 3 03/19/2024 3 03/18/2024 5 03/17/2024 4 03/16/2024 3 03/15/2024 4 03/14/2024 3 03/13/2024 2 03/12/2024 3 03/11/2024 3 03/10/2024 1 03/09/2024 3 03/08/2024 2 03/07/2024 3 ```

MrUnknownDE closed this issue

2026-04-06 01:41:22 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/vaultwarden#1172