Slack Integration Rate Limiting - Channel Dropdown Empty and Test Rule Fails #431

New Issue

MrUnknownDE · 2026-04-05T16:19:44+02:00

MrUnknownDE commented

2026-04-05 16:19:44 +02:00

Originally created by @bourchi-ali on 6/26/2025

Slack Rate Limiting Issue Persists After Fix

Bug Description

The Slack integration rate limiting issue reported in #1948 and addressed in commit [8765bc0](8765bc07ed) is still occurring, though the error location has shifted from line 213 to line 262.

Current Status

Previous Issue: Bump engine.io from 6.4.1 to 6.4.2 in /ProbeAPI (#1948)
Previous Fix: 8765bc0
Current Version: Latest release (post-upgrade)
Status: Issue persists with different error location

Problem Details

Symptoms

Channel Dropdown Empty: Slack channel dropdown remains empty when creating notification rules
Test Rule Failures: Test Rule functionality fails with rate limiting errors
Integration Broken: Two-way Slack integration features (slash commands, interactive features) not working

Error Details

BadDataException [Error]: Cannot post message to channel. ratelimited
at Service.testRule (/usr/src/Common/Server/Services/WorkspaceNotificationRuleService.ts:262:17)

Full API Response

HTTPErrorResponse {
  "_statusCode": 429,
  "_headers": {
    "retry-after": "30",
    "x-oauth-scopes": "commands,channels:history,channels:join,channels:manage,channels:read,channels:write.invites,channels:write.topic,chat:write,users:read,groups:read,groups:write,im:read,im:write,mpim:read,mpim:write"
  },
  "_data": { 
    "ok": false, 
    "error": "ratelimited" 
  }
}

Root Cause Analysis

The previous fix appears to have been incomplete. While it may have addressed rate limiting in specific scenarios, it didn't implement comprehensive rate limiting protection across all Slack API operations. The core issue of excessive API calls without proper throttling remains unresolved.

Steps to Reproduce

Navigate to notification rules creation
Attempt to select Slack channel from dropdown
Observe empty dropdown
Try to test the rule
Receive rate limiting error

Expected Behavior

Channel dropdown should populate with available Slack channels
Test Rule should successfully send test notifications
Two-way Slack integration should function properly
API calls should be properly throttled to respect Slack's rate limits

Proposed Solution

Implement comprehensive rate limiting across all Slack API endpoints, not just specific scenarios:

Global Rate Limiting: Implement rate limiting wrapper for all Slack API calls
Retry Logic: Add exponential backoff with jitter for 429 responses
Request Queuing: Queue requests when approaching rate limits
Caching: Cache channel lists and other frequently accessed data

Environment

Deployment: Docker Compose
Slack App: Properly configured with all required permissions

Request for Information

Can other users confirm if they're experiencing similar issues with the latest release? This will help determine if it's a widespread problem or environment-specific.

*Originally created by @bourchi-ali on 6/26/2025* # Slack Rate Limiting Issue Persists After Fix ## Bug Description The Slack integration rate limiting issue reported in #1948 and addressed in commit [[8765bc0](https://github.com/OneUptime/oneuptime/commit/8765bc07ed6aa10dd562cd4f8f5fd6dd0aaa3a63)](https://github.com/OneUptime/oneuptime/commit/8765bc07ed6aa10dd562cd4f8f5fd6dd0aaa3a63) is still occurring, though the error location has shifted from line 213 to line 262. ## Current Status - **Previous Issue**: #1948 - **Previous Fix**: 8765bc0 - **Current Version**: Latest release (post-upgrade) - **Status**: Issue persists with different error location ## Problem Details ### Symptoms 1. **Channel Dropdown Empty**: Slack channel dropdown remains empty when creating notification rules 2. **Test Rule Failures**: Test Rule functionality fails with rate limiting errors 3. **Integration Broken**: Two-way Slack integration features (slash commands, interactive features) not working ### Error Details ``` BadDataException [Error]: Cannot post message to channel. ratelimited at Service.testRule (/usr/src/Common/Server/Services/WorkspaceNotificationRuleService.ts:262:17) ``` ### Full API Response ```json HTTPErrorResponse { "_statusCode": 429, "_headers": { "retry-after": "30", "x-oauth-scopes": "commands,channels:history,channels:join,channels:manage,channels:read,channels:write.invites,channels:write.topic,chat:write,users:read,groups:read,groups:write,im:read,im:write,mpim:read,mpim:write" }, "_data": { "ok": false, "error": "ratelimited" } } ``` ## Root Cause Analysis The previous fix appears to have been **incomplete**. While it may have addressed rate limiting in specific scenarios, it didn't implement comprehensive rate limiting protection across all Slack API operations. The core issue of excessive API calls without proper throttling remains unresolved. ## Steps to Reproduce 1. Navigate to notification rules creation 2. Attempt to select Slack channel from dropdown 3. Observe empty dropdown 4. Try to test the rule 5. Receive rate limiting error ## Expected Behavior - Channel dropdown should populate with available Slack channels - Test Rule should successfully send test notifications - Two-way Slack integration should function properly - API calls should be properly throttled to respect Slack's rate limits ## Proposed Solution Implement comprehensive rate limiting across **all** Slack API endpoints, not just specific scenarios: 1. **Global Rate Limiting**: Implement rate limiting wrapper for all Slack API calls 2. **Retry Logic**: Add exponential backoff with jitter for 429 responses 3. **Request Queuing**: Queue requests when approaching rate limits 4. **Caching**: Cache channel lists and other frequently accessed data ## Environment - Deployment: Docker Compose - Slack App: Properly configured with all required permissions ## Request for Information Can other users confirm if they're experiencing similar issues with the latest release? This will help determine if it's a widespread problem or environment-specific.

MrUnknownDE added the bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug bug labels 2026-04-05 16:19:44 +02:00

Sign in to join this conversation.

Branches Tags

master

release

snyk-fix-0aa85871afc5c8eee83a20407bd9e063

new-dash

probe-runner

better-logs

merge-ui

api-ref-refactor

safe-vm

tf-demo

dash-chunk

mult-slack

episode-resolve

feat/homepage-seo-v2

seo/compare-pages-homepage-optimize

tf-sets

vite-build

alert-grouping-product

oneuptime-fix-exception-5f43cb12

deprecate-telemetry-service

oneuptime-fix-exception-b77cfaf1

workflow-ui-update

chore/bump-version-9.2.5-1764859493

chore/bump-version-9.2.5-1764857051

state-change

postmortem

chore/bump-version-9.1.3-1764685359

pricing-compare

use-cases

dashboard-logs

incident-declared-at

agent-lib

scim-patch-fix

files-notes

dropdown-labels

revoke-token

refresh-sessions

oneuptime-ssl

telemetry-attr

nativewind

oneuptime-mobile-app

timepkr

cert-kube

new-log-ui

ms-temas-bot

span-improve

teams-integration

cert-manager

final-ms-teams

new-markdown

scim-update

probe-proxy-impl

probe-proxy-url

fb-whatsapp

whatsapp-integration

ms-teams-conn

copilot-plus

vs-code-copilot

ms-teams

server-bucket-size

sp-scim

scim

probe-queue

queue-work

keda-autoscaler

otel-keda

telemetry-queue

otel-queue

mobile-responsive

add-state-to-title

file-tf

snyk-fix-21d325191a83f6e495f02680309be6df

new-terraform-provider

tf-provider

msp-server

announcement-templates

optmize-esbuild

ms-teams-workspace-integration

run-as-non-root

fix/team-member-service-test-error

feature/msteams-integration-backend

optmize-dash

split-dashboard

feature/responsive-navbar

feat/mobile-responsive-sidemenu

mobile-navbar

mobile-ui

fix-recurring-events

fix-monitor-test

snyk-fix-6d5b446d98eb30ad7cc48664f26e8dd3

snyk-fix-5f2a25c13d925a6df331a8db1b4b6bc3

snyk-fix-82cf608c1e34aadeec1e40be2dd7692c

snyk-fix-28fb72b74f8e210b21994c9d4a749db4

snyk-fix-9dd32ce4170d36a0032a14eca7bcc974

snyk-fix-33cfbfb6119028a909c55ae0b6f42134

snyk-fix-0d6c626395070c0bf0647893fd4ad82c

telemetry-decorator

snyk-fix-9dcd00f3db77cda7a8a956ad4c744638

snyk-fix-39e89b36dc9e401d2beb27cf134cb8f8

manual-workflow

sp-api

snyk-fix-476babcac00c78674ae2231bc91de4f0

snyk-fix-aba76b601f35a212295c529b373457fa

slack-app

otel-collector-queue-size

scheduled-maintenance-feed

add-proxy-to-server-monitor

snyk-fix-129563dbc737d32bc6bb8e3311b8bb86

metrics-monitor

snyk-fix-b3faaf7e1b0131c254a19d461fdfa12f

snyk-fix-49a541156a8e299f9158ecc2e1ce3e4e

snyk-fix-6fed1c2110e88e596893df3351740515

snyk-fix-7f3c8e70773dfe11ea8caed8e34b2573

snyk-fix-2643b617b033d0cbf19291b6ad7776a4

snyk-fix-1219e1241120fcf13feb7bc0b54e18e3

merge-monitor-metrics

dashboarding

oneuptime-copilot-oneuptime-copilot-improve-comments-ZlPvy

oneuptime-copilot-oneuptime-copilot-improve-comments-vThJD

oneuptime-copilot-oneuptime-copilot-improve-comments-veTFM

snyk-fix-1230088cdc5e6714c07c4043881095db

sp-custom-cert

realtime

snyk-fix-2267fc7afa5dd1aadc03bda66d835a26

snyk-fix-82eb97bf3553f75fcdec629b2375d091

snyk-fix-0ff1eeedd26324c34c4b6446c78d93ab

snyk-fix-7a5cb77d5e30b278d6086cc6ec7563dd

snyk-fix-80a68103cb676782e141f18725deae71

oneuptime-copilot-oneuptime-copilot-add-spans-Dcefh

oneuptime-copilot-oneuptime-copilot-improve-comments-YRJKG

oneuptime-copilot-oneuptime-copilot-improve-comments-PBqVE

snyk-fix-6390d5be9d2dcbe507da6d19b87bb581

oneuptime-copilot-oneuptime-copilot-improve-comments-dnfTZ

copilot-main

error-tracking

snyk-fix-616369b0f1423d8113422a2b0ea1798a

snyk-fix-ecde2433ac879fc83fcb7fbaef849ca2

snyk-fix-1436684cb4b974b4b38ec0de7c27678b

snyk-fix-929b1416ccb03655ff400075fc126a5d

merge-dir

snyk-upgrade-bc55cc4a3bed3c919d5253c5ccb89569

oneuptime-copilot-oneuptime-copilot-improve-comments-NXaNT

oneuptime-copilot-oneuptime-copilot-improve-comments-MhFUX

oneuptime-copilot-oneuptime-copilot-improve-comments-eHoNt

oneuptime-copilot-oneuptime-copilot-improve-comments-BWpyo

landing-page

clickhouse-optimization

attributes-cache

snyk-upgrade-b1d7fc785c4b347f9540423872d4f7eb

snyk-upgrade-b12f9d5ede2ce54b392c16e1398efc0d

snyk-upgrade-90863abe750274199042129d216ec458

snyk-upgrade-24742a40cc66c0ff20127c49b31c5b92

status-page-reports

monitor-reports

oneuptime-app-oneuptime-app-improve-comments-wPkHZ

oneuptime-app-oneuptime-app-improve-comments-zheFq

oneuptime-app-oneuptime-app-improve-comments-yGblU

oneuptime-app-oneuptime-app-improve-comments-XvJyI

oneuptime-app-oneuptime-app-improve-comments-Iigpp

oneuptime-app-oneuptime-app-improve-comments-BLqEd

oneuptime-app-oneuptime-app-improve-comments-BSWXD

oneuptime-app-oneuptime-app-improve-comments-iDNIR

oneuptime-app-oneuptime-app-improve-comments-FPiCN

oneuptime-app-oneuptime-app-improve-comments-hfdRa

oneuptime-app-oneuptime-app-improve-comments-yEhKY

oneuptime-app-oneuptime-app-improve-comments-iOjcB

oneuptime-app-oneuptime-app-improve-comments-tFtyx

oneuptime-app-oneuptime-app-improve-comments-sBvxI

oneuptime-app-oneuptime-app-improve-comments-IXAhB

oneuptime-app-oneuptime-app-improve-comments-YtccK

oneuptime-app-oneuptime-app-improve-comments-QCnqA

oneuptime-app-oneuptime-app-improve-comments-NMjBi

oneuptime-app-oneuptime-app-improve-comments-lDSQP

oneuptime-app-oneuptime-app-improve-comments-YdAAE

oneuptime-app-oneuptime-app-improve-comments-JVpQw

oneuptime-app-oneuptime-app-improve-comments-DncIm

oneuptime-app-oneuptime-app-improve-comments-iBzpC

oneuptime-app-oneuptime-app-improve-comments-GkOnn

telemetry-backend

snyk-upgrade-818024a85e0cd5b743112ffb7f513123

snyk-upgrade-2f9ef5e24aa83032721fce51c3a0b0ca

oneuptime-app-oneuptime-app-refactor-code-nDOMw

oneuptime-app-oneuptime-app-refactor-code-nzaSs

oneuptime-app-oneuptime-app-refactor-code-TIxRv

oneuptime-app-oneuptime-app-refactor-code-Llyxi

oneuptime-app-oneuptime-app-refactor-code-lfIez

oneuptime-app-oneuptime-app-improve-comments-FLlpf

oneuptime-app-oneuptime-app-improve-comments-DaGfY

oneuptime-app-oneuptime-app-improve-comments-XaYkK

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-CDFoR

ai-copilot-main

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-hueYU

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-HQfro

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-JVpKF

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-mSRcD

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-NXSdG

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-CYFVx

oneuptime-app-oneuptime-app-improve-comments-UsHIA

oneuptime-app-oneuptime-app-improve-comments-ZBFjb

oneuptime-app-oneuptime-app-improve-comments-ggYpS

oneuptime-app-oneuptime-app-improve-comments-YUFeh

eval-over-time

oneuptime-app-oneuptime-app-improve-comments-LkJYk

ai-main-70b

eslint

oneuptime-app-test-branch-11

oneuptime-app-test-branch-10

oneuptime-app-test-branch-7

oneuptime-app-test-branch-6

test-branch

nz/metric-detail

ext-postgres

perms-refactor

filter-refactor

group-by

greenlock-refactor

attributes

react-app

otel

snyk-upgrade-3677f1da0f20b9341d779856ea606cce

snyk-upgrade-324862099bddfa90e5e0e06eb934c162

renovate/node-16.x-lockfile

snyk-upgrade-ac717665d10b7db0ddb20de9bdaf435b

snyk-upgrade-7b7ea7a1c722d2b7f48e290063771f7b

snyk-upgrade-520b5a017b5e7985121ef90b57317068

renovate/stripe-react-stripe-js-1.x-lockfile

renovate/heroicons-react-2.x-lockfile

renovate/headlessui-react-1.x-lockfile

renovate/clickhouse-clickhouse-server-23.x

snyk-upgrade-8a1c1f353568dcbc7b2c3e4beec4a6b7

snyk-upgrade-affb0a1678b57ae5a6f3183f5e8d302d

renovate/vm2-3.x-lockfile

renovate/start-server-and-test-1.x-lockfile

renovate/chrome-launcher-0.x-lockfile

renovate/mikepenz-release-changelog-builder-action-3.x

renovate/npm-xmldom-vulnerability

monitor-groups

feature-flags

logging

admin-dashboard

update-deps

metered-plan

fix-status-page-refresh

owner-emails

owners-in-form

monitoring

fix-small-issues

single-docker-container

workflow-fix

staus-page-fix

slack_workflow

emails

fix/ping-monitor-ipv6

fix-typescript-errors

issue/160-add-setup-alert

143-fix-github-actions-jobs-dont-run-on-prs-from-forked-repo

142-github-actions-build-job-to-run-home-tests

147-form-containers

131-add-styling-for-account-in-topbar

chore/setup-sonar-cloud-#21

96-css-for-login-page

97-css-login-page

35-write-docker-build-job-for-helmchart

ts-migrate

monitor-box

feature-github-actions-3

feature-github-actions

feature-github-actions-2

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/oneuptime#431