Provider returns label arrays in non-deterministic order causing inconsistent result errors #140

New Issue

MrUnknownDE · 2026-04-05T16:18:56+02:00

MrUnknownDE commented

2026-04-05 16:18:56 +02:00

Originally created by @listellm on 1/23/2026

Summary

Provider returns label arrays in non-deterministic order, causing "Provider produced inconsistent result after apply" errors and perpetual drift detection.

Problem Description

When updating a resource with labels (e.g., oneuptime_probe), the provider successfully applies the changes but fails Terraform's round-trip validation because the API returns labels in a different order than specified in the configuration.

Symptoms

Error: Provider produced inconsistent result after apply
Labels[0] and labels[1] swap positions between apply and read
Resource is marked as tainted on every apply
Terraform tries to recreate the resource unnecessarily

Root Cause Analysis

Provider Code Location

Terraform/terraform-provider-oneuptime/internal/provider/resource_probe.go

Lines 352-377 (Create), 659-684 (Read), 1024-1049 (Update):

if val, ok := dataMap["labels"].([]interface{}); ok {
    // Convert API response list to Terraform list
    var listItems []attr.Value
    for _, item := range val {
        if itemMap, ok := item.(map[string]interface{}); ok {
            if id, ok := itemMap["_id"].(string); ok {
                listItems = append(listItems, types.StringValue(id))
            }
        }
    }
    data.Labels = types.ListValueMust(types.StringType, listItems)
}

Issue: The provider appends labels in whatever order the API returns them, without any sorting or normalization.

API Behavior

The OneUptime API does not guarantee label order preservation. Labels are likely returned in:

Database insertion order
Internal ID order
Non-deterministic order

Terraform Expectation

Terraform compares list values by position:

labels[0] in config must equal labels[0] from API
labels[1] in config must equal labels[1] from API
etc.

Steps to Reproduce

Create a probe with multiple labels:

resource "oneuptime_probe" "example" {
  name         = "example-probe"
  project_id   = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  key          = "example-key"
  probe_version = "9.4.7"
  labels = [
    "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", # cloud:aws
    "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy", # environment:dev
    "zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz", # region:eu-central-1
    "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", # server:cluster-01
  ]
}

Run terraform apply
Observe error:

Error: Provider produced inconsistent result after apply

When applying changes to oneuptime_probe.example, provider
"provider[\"registry.terraform.io/oneuptime/oneuptime\"]" produced an
unexpected new value: .labels[0]: was
cty.StringVal("xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"), but now
cty.StringVal("yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy").

Check state: Labels exist but in different order
Run terraform apply again: Same error repeats

Expected Behavior

Labels should be returned in the same order as sent
OR labels should be sorted consistently (e.g., alphabetically by ID)
No drift should be detected when label content is identical

Actual Behavior

Labels are returned in API-determined order
Order differs from configuration order
Terraform detects drift on every apply
Resource is unnecessarily recreated

Impact

Severity: High
Affects: All resources with label arrays (oneuptime_probe, oneuptime_monitor, oneuptime_status_page, etc.)
Workaround: Lifecycle ignore_changes = [labels] (not ideal, loses drift detection)

Proposed Fix

Option 1: Sort in Provider (Recommended)

Sort label arrays after reading from API to ensure deterministic order:

if val, ok := dataMap["labels"].([]interface{}); ok {
    var listItems []attr.Value
    var labelIDs []string

    // Extract all label IDs
    for _, item := range val {
        if itemMap, ok := item.(map[string]interface{}); ok {
            if id, ok := itemMap["_id"].(string); ok {
                labelIDs = append(labelIDs, id)
            }
        }
    }

    // Sort alphabetically for consistent ordering
    sort.Strings(labelIDs)

    // Convert to Terraform values
    for _, id := range labelIDs {
        listItems = append(listItems, types.StringValue(id))
    }

    data.Labels = types.ListValueMust(types.StringType, listItems)
}

Option 2: Use TypeSet Instead of TypeList

Change schema to use types.SetType which is order-independent:

"labels": schema.SetAttribute{
    ElementType: types.StringType,
    // ...
}

Recommendation: Option 1 (sorting) is safer as it maintains backwards compatibility with existing state files.

#2228 - Provider returned wrapper objects instead of plain strings (fixed in v9.4.7)
#2230 - ResourceGenerator complex object regression (fixed in v9.4.7)

All three issues share the same root category: round-trip consistency failures between provider CREATE/UPDATE and READ operations.

Environment

Provider Version: 9.4.7
Terraform Version: 1.12.0
Affected Resources: oneuptime_probe, oneuptime_monitor, oneuptime_status_page, and any resource with label arrays

Additional Context

The issue occurs because:

CREATE/UPDATE sends labels as [{\"_id\": \"uuid1\"}, {\"_id\": \"uuid2\"}]
API stores labels in database
READ returns labels as [{\"_id\": \"uuid2\"}, {\"_id\": \"uuid1\"}] (different order)
Provider doesn't normalize order
Terraform detects drift due to positional mismatch

*Originally created by @listellm on 1/23/2026* ## Summary Provider returns label arrays in non-deterministic order, causing "Provider produced inconsistent result after apply" errors and perpetual drift detection. ## Problem Description When updating a resource with labels (e.g., `oneuptime_probe`), the provider successfully applies the changes but fails Terraform's round-trip validation because the API returns labels in a different order than specified in the configuration. ## Symptoms - Error: `Provider produced inconsistent result after apply` - Labels[0] and labels[1] swap positions between apply and read - Resource is marked as tainted on every apply - Terraform tries to recreate the resource unnecessarily ## Root Cause Analysis ### Provider Code Location `Terraform/terraform-provider-oneuptime/internal/provider/resource_probe.go` Lines 352-377 (Create), 659-684 (Read), 1024-1049 (Update): ```go if val, ok := dataMap["labels"].([]interface{}); ok { // Convert API response list to Terraform list var listItems []attr.Value for _, item := range val { if itemMap, ok := item.(map[string]interface{}); ok { if id, ok := itemMap["_id"].(string); ok { listItems = append(listItems, types.StringValue(id)) } } } data.Labels = types.ListValueMust(types.StringType, listItems) } ``` **Issue**: The provider appends labels in whatever order the API returns them, without any sorting or normalization. ### API Behavior The OneUptime API does not guarantee label order preservation. Labels are likely returned in: - Database insertion order - Internal ID order - Non-deterministic order ### Terraform Expectation Terraform compares list values by position: - `labels[0]` in config must equal `labels[0]` from API - `labels[1]` in config must equal `labels[1]` from API - etc. ## Steps to Reproduce 1. Create a probe with multiple labels: ```hcl resource "oneuptime_probe" "example" { name = "example-probe" project_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" key = "example-key" probe_version = "9.4.7" labels = [ "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", # cloud:aws "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy", # environment:dev "zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz", # region:eu-central-1 "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", # server:cluster-01 ] } ``` 2. Run `terraform apply` 3. Observe error: ``` Error: Provider produced inconsistent result after apply When applying changes to oneuptime_probe.example, provider "provider[\"registry.terraform.io/oneuptime/oneuptime\"]" produced an unexpected new value: .labels[0]: was cty.StringVal("xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"), but now cty.StringVal("yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"). ``` 4. Check state: Labels exist but in different order 5. Run `terraform apply` again: Same error repeats ## Expected Behavior - Labels should be returned in the same order as sent - OR labels should be sorted consistently (e.g., alphabetically by ID) - No drift should be detected when label content is identical ## Actual Behavior - Labels are returned in API-determined order - Order differs from configuration order - Terraform detects drift on every apply - Resource is unnecessarily recreated ## Impact - **Severity**: High - **Affects**: All resources with label arrays (`oneuptime_probe`, `oneuptime_monitor`, `oneuptime_status_page`, etc.) - **Workaround**: Lifecycle `ignore_changes = [labels]` (not ideal, loses drift detection) ## Proposed Fix ### Option 1: Sort in Provider (Recommended) Sort label arrays after reading from API to ensure deterministic order: ```go if val, ok := dataMap["labels"].([]interface{}); ok { var listItems []attr.Value var labelIDs []string // Extract all label IDs for _, item := range val { if itemMap, ok := item.(map[string]interface{}); ok { if id, ok := itemMap["_id"].(string); ok { labelIDs = append(labelIDs, id) } } } // Sort alphabetically for consistent ordering sort.Strings(labelIDs) // Convert to Terraform values for _, id := range labelIDs { listItems = append(listItems, types.StringValue(id)) } data.Labels = types.ListValueMust(types.StringType, listItems) } ``` ### Option 2: Use TypeSet Instead of TypeList Change schema to use `types.SetType` which is order-independent: ```go "labels": schema.SetAttribute{ ElementType: types.StringType, // ... } ``` **Recommendation**: Option 1 (sorting) is safer as it maintains backwards compatibility with existing state files. ## Related Issues - #2228 - Provider returned wrapper objects instead of plain strings (fixed in v9.4.7) - #2230 - ResourceGenerator complex object regression (fixed in v9.4.7) All three issues share the same root category: **round-trip consistency failures** between provider CREATE/UPDATE and READ operations. ## Environment - **Provider Version**: 9.4.7 - **Terraform Version**: 1.12.0 - **Affected Resources**: `oneuptime_probe`, `oneuptime_monitor`, `oneuptime_status_page`, and any resource with label arrays ## Additional Context The issue occurs because: 1. CREATE/UPDATE sends labels as `[{\"_id\": \"uuid1\"}, {\"_id\": \"uuid2\"}]` 2. API stores labels in database 3. READ returns labels as `[{\"_id\": \"uuid2\"}, {\"_id\": \"uuid1\"}]` (different order) 4. Provider doesn't normalize order 5. Terraform detects drift due to positional mismatch

MrUnknownDE closed this issue

2026-04-05 16:18:56 +02:00

Sign in to join this conversation.

Branches Tags

master

release

snyk-fix-0aa85871afc5c8eee83a20407bd9e063

new-dash

probe-runner

better-logs

merge-ui

api-ref-refactor

safe-vm

tf-demo

dash-chunk

mult-slack

episode-resolve

feat/homepage-seo-v2

seo/compare-pages-homepage-optimize

tf-sets

vite-build

alert-grouping-product

oneuptime-fix-exception-5f43cb12

deprecate-telemetry-service

oneuptime-fix-exception-b77cfaf1

workflow-ui-update

chore/bump-version-9.2.5-1764859493

chore/bump-version-9.2.5-1764857051

state-change

postmortem

chore/bump-version-9.1.3-1764685359

pricing-compare

use-cases

dashboard-logs

incident-declared-at

agent-lib

scim-patch-fix

files-notes

dropdown-labels

revoke-token

refresh-sessions

oneuptime-ssl

telemetry-attr

nativewind

oneuptime-mobile-app

timepkr

cert-kube

new-log-ui

ms-temas-bot

span-improve

teams-integration

cert-manager

final-ms-teams

new-markdown

scim-update

probe-proxy-impl

probe-proxy-url

fb-whatsapp

whatsapp-integration

ms-teams-conn

copilot-plus

vs-code-copilot

ms-teams

server-bucket-size

sp-scim

scim

probe-queue

queue-work

keda-autoscaler

otel-keda

telemetry-queue

otel-queue

mobile-responsive

add-state-to-title

file-tf

snyk-fix-21d325191a83f6e495f02680309be6df

new-terraform-provider

tf-provider

msp-server

announcement-templates

optmize-esbuild

ms-teams-workspace-integration

run-as-non-root

fix/team-member-service-test-error

feature/msteams-integration-backend

optmize-dash

split-dashboard

feature/responsive-navbar

feat/mobile-responsive-sidemenu

mobile-navbar

mobile-ui

fix-recurring-events

fix-monitor-test

snyk-fix-6d5b446d98eb30ad7cc48664f26e8dd3

snyk-fix-5f2a25c13d925a6df331a8db1b4b6bc3

snyk-fix-82cf608c1e34aadeec1e40be2dd7692c

snyk-fix-28fb72b74f8e210b21994c9d4a749db4

snyk-fix-9dd32ce4170d36a0032a14eca7bcc974

snyk-fix-33cfbfb6119028a909c55ae0b6f42134

snyk-fix-0d6c626395070c0bf0647893fd4ad82c

telemetry-decorator

snyk-fix-9dcd00f3db77cda7a8a956ad4c744638

snyk-fix-39e89b36dc9e401d2beb27cf134cb8f8

manual-workflow

sp-api

snyk-fix-476babcac00c78674ae2231bc91de4f0

snyk-fix-aba76b601f35a212295c529b373457fa

slack-app

otel-collector-queue-size

scheduled-maintenance-feed

add-proxy-to-server-monitor

snyk-fix-129563dbc737d32bc6bb8e3311b8bb86

metrics-monitor

snyk-fix-b3faaf7e1b0131c254a19d461fdfa12f

snyk-fix-49a541156a8e299f9158ecc2e1ce3e4e

snyk-fix-6fed1c2110e88e596893df3351740515

snyk-fix-7f3c8e70773dfe11ea8caed8e34b2573

snyk-fix-2643b617b033d0cbf19291b6ad7776a4

snyk-fix-1219e1241120fcf13feb7bc0b54e18e3

merge-monitor-metrics

dashboarding

oneuptime-copilot-oneuptime-copilot-improve-comments-ZlPvy

oneuptime-copilot-oneuptime-copilot-improve-comments-vThJD

oneuptime-copilot-oneuptime-copilot-improve-comments-veTFM

snyk-fix-1230088cdc5e6714c07c4043881095db

sp-custom-cert

realtime

snyk-fix-2267fc7afa5dd1aadc03bda66d835a26

snyk-fix-82eb97bf3553f75fcdec629b2375d091

snyk-fix-0ff1eeedd26324c34c4b6446c78d93ab

snyk-fix-7a5cb77d5e30b278d6086cc6ec7563dd

snyk-fix-80a68103cb676782e141f18725deae71

oneuptime-copilot-oneuptime-copilot-add-spans-Dcefh

oneuptime-copilot-oneuptime-copilot-improve-comments-YRJKG

oneuptime-copilot-oneuptime-copilot-improve-comments-PBqVE

snyk-fix-6390d5be9d2dcbe507da6d19b87bb581

oneuptime-copilot-oneuptime-copilot-improve-comments-dnfTZ

copilot-main

error-tracking

snyk-fix-616369b0f1423d8113422a2b0ea1798a

snyk-fix-ecde2433ac879fc83fcb7fbaef849ca2

snyk-fix-1436684cb4b974b4b38ec0de7c27678b

snyk-fix-929b1416ccb03655ff400075fc126a5d

merge-dir

snyk-upgrade-bc55cc4a3bed3c919d5253c5ccb89569

oneuptime-copilot-oneuptime-copilot-improve-comments-NXaNT

oneuptime-copilot-oneuptime-copilot-improve-comments-MhFUX

oneuptime-copilot-oneuptime-copilot-improve-comments-eHoNt

oneuptime-copilot-oneuptime-copilot-improve-comments-BWpyo

landing-page

clickhouse-optimization

attributes-cache

snyk-upgrade-b1d7fc785c4b347f9540423872d4f7eb

snyk-upgrade-b12f9d5ede2ce54b392c16e1398efc0d

snyk-upgrade-90863abe750274199042129d216ec458

snyk-upgrade-24742a40cc66c0ff20127c49b31c5b92

status-page-reports

monitor-reports

oneuptime-app-oneuptime-app-improve-comments-wPkHZ

oneuptime-app-oneuptime-app-improve-comments-zheFq

oneuptime-app-oneuptime-app-improve-comments-yGblU

oneuptime-app-oneuptime-app-improve-comments-XvJyI

oneuptime-app-oneuptime-app-improve-comments-Iigpp

oneuptime-app-oneuptime-app-improve-comments-BLqEd

oneuptime-app-oneuptime-app-improve-comments-BSWXD

oneuptime-app-oneuptime-app-improve-comments-iDNIR

oneuptime-app-oneuptime-app-improve-comments-FPiCN

oneuptime-app-oneuptime-app-improve-comments-hfdRa

oneuptime-app-oneuptime-app-improve-comments-yEhKY

oneuptime-app-oneuptime-app-improve-comments-iOjcB

oneuptime-app-oneuptime-app-improve-comments-tFtyx

oneuptime-app-oneuptime-app-improve-comments-sBvxI

oneuptime-app-oneuptime-app-improve-comments-IXAhB

oneuptime-app-oneuptime-app-improve-comments-YtccK

oneuptime-app-oneuptime-app-improve-comments-QCnqA

oneuptime-app-oneuptime-app-improve-comments-NMjBi

oneuptime-app-oneuptime-app-improve-comments-lDSQP

oneuptime-app-oneuptime-app-improve-comments-YdAAE

oneuptime-app-oneuptime-app-improve-comments-JVpQw

oneuptime-app-oneuptime-app-improve-comments-DncIm

oneuptime-app-oneuptime-app-improve-comments-iBzpC

oneuptime-app-oneuptime-app-improve-comments-GkOnn

telemetry-backend

snyk-upgrade-818024a85e0cd5b743112ffb7f513123

snyk-upgrade-2f9ef5e24aa83032721fce51c3a0b0ca

oneuptime-app-oneuptime-app-refactor-code-nDOMw

oneuptime-app-oneuptime-app-refactor-code-nzaSs

oneuptime-app-oneuptime-app-refactor-code-TIxRv

oneuptime-app-oneuptime-app-refactor-code-Llyxi

oneuptime-app-oneuptime-app-refactor-code-lfIez

oneuptime-app-oneuptime-app-improve-comments-FLlpf

oneuptime-app-oneuptime-app-improve-comments-DaGfY

oneuptime-app-oneuptime-app-improve-comments-XaYkK

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-CDFoR

ai-copilot-main

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-hueYU

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-HQfro

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-JVpKF

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-mSRcD

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-NXSdG

oneuptime-app-oneuptime-app-fix-grammar-and-spelling-CYFVx

oneuptime-app-oneuptime-app-improve-comments-UsHIA

oneuptime-app-oneuptime-app-improve-comments-ZBFjb

oneuptime-app-oneuptime-app-improve-comments-ggYpS

oneuptime-app-oneuptime-app-improve-comments-YUFeh

eval-over-time

oneuptime-app-oneuptime-app-improve-comments-LkJYk

ai-main-70b

eslint

oneuptime-app-test-branch-11

oneuptime-app-test-branch-10

oneuptime-app-test-branch-7

oneuptime-app-test-branch-6

test-branch

nz/metric-detail

ext-postgres

perms-refactor

filter-refactor

group-by

greenlock-refactor

attributes

react-app

otel

snyk-upgrade-3677f1da0f20b9341d779856ea606cce

snyk-upgrade-324862099bddfa90e5e0e06eb934c162

renovate/node-16.x-lockfile

snyk-upgrade-ac717665d10b7db0ddb20de9bdaf435b

snyk-upgrade-7b7ea7a1c722d2b7f48e290063771f7b

snyk-upgrade-520b5a017b5e7985121ef90b57317068

renovate/stripe-react-stripe-js-1.x-lockfile

renovate/heroicons-react-2.x-lockfile

renovate/headlessui-react-1.x-lockfile

renovate/clickhouse-clickhouse-server-23.x

snyk-upgrade-8a1c1f353568dcbc7b2c3e4beec4a6b7

snyk-upgrade-affb0a1678b57ae5a6f3183f5e8d302d

renovate/vm2-3.x-lockfile

renovate/start-server-and-test-1.x-lockfile

renovate/chrome-launcher-0.x-lockfile

renovate/mikepenz-release-changelog-builder-action-3.x

renovate/npm-xmldom-vulnerability

monitor-groups

feature-flags

logging

admin-dashboard

update-deps

metered-plan

fix-status-page-refresh

owner-emails

owners-in-form

monitoring

fix-small-issues

single-docker-container

workflow-fix

staus-page-fix

slack_workflow

emails

fix/ping-monitor-ipv6

fix-typescript-errors

issue/160-add-setup-alert

143-fix-github-actions-jobs-dont-run-on-prs-from-forked-repo

142-github-actions-build-job-to-run-home-tests

147-form-containers

131-add-styling-for-account-in-topbar

chore/setup-sonar-cloud-#21

96-css-for-login-page

97-css-login-page

35-write-docker-build-job-for-helmchart

ts-migrate

monitor-box

feature-github-actions-3

feature-github-actions

feature-github-actions-2

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/oneuptime#140