Show partial script output during job execution (not just after completion) #92

New Issue

MrUnknownDE · 2026-04-05T16:21:35+02:00

MrUnknownDE commented

2026-04-05 16:21:35 +02:00

Originally created by @berlikm on 3/24/2026

NetBox version

v4.5.3

Feature type

Change to existing functionality

Proposed functionality

NetBox version

4.5.x

Feature type

Enhancement

Proposed functionality

Currently, the script result page (extras/htmx/script_result.html) only renders log output after the job completes:

{% if job.completed %}
    {# Render log entries, output, tests #}
{% elif job.started %}
    {% include 'extras/inc/result_pending.html' %}
{% endif %}

This means users see only a spinner and "Results pending..." during the entire execution, even though self.log_info() calls accumulate data in job.data throughout the run.

For long-running scripts (e.g., firmware upgrades across dozens of network devices taking 10–30+ minutes), this provides no feedback on progress. Users cannot tell if the script is working, stuck, or how far along it is.

Proposed change

Modify the result_pending.html include (or the conditional in script_result.html) to render log entries from job.data.log while the job is running. The HTMX auto-refresh (every 5s) is already in place, so the UI would naturally update as new entries appear.

A minimal template change could be:

{% elif job.started %}
    {% include 'extras/inc/result_pending.html' %}
    {% if job.data.log %}
        <div class="mt-3">
            <table class="table table-hover">
                {% for entry in job.data.log %}
                <tr>
                    <td><small class="text-muted">{{ entry.time|isodatetime }}</small></td>
                    <td>{% badge entry.status %}</td>
                    <td>{{ entry.message|markdown }}</td>
                </tr>
                {% endfor %}
            </table>
        </div>
    {% endif %}
{% endif %}

However, this alone is insufficient because job.data is only written to the database when the script completes (in the ScriptJob.run_script() finally block). The data needs to be committed during execution for the HTMX refresh to pick it up.

Why intermediate saves are hard

ScriptJob.run_script() wraps execution in transaction.atomic(), so any intermediate job.save() within the script is invisible to the web frontend (different DB connection, READ COMMITTED transaction isolation).

Attempting to bypass the transaction with a separate database connection (e.g., raw SQL via psycopg) causes the new connection to block on a row-level lock held by the script's transaction, effectively freezing the script.

Possible approaches

Periodically flush job.data outside the atomic block — e.g., using a SAVEPOINT + partial commit, or by restructuring the transaction boundaries.
Use a side channel (Django cache, Redis pub/sub via RQ, or a separate progress table not locked by the transaction) to stream progress data to the frontend.
Write to job.log_entries (from #19816) via a non-transactional path and render those entries in the pending state template.
Move the job.data save into the script loop by providing a flush() or save_progress() method on the Script class that commits data to a separate, non-transactional table or cache.

Current workaround

Script authors can write progress to a log file on the server (e.g., /var/log/) and monitor with tail -f. The _job_log() entries (via job.log_entries) also appear in the Log tab after the job completes, but not during execution.

Use case

Network automation: Scripts upgrading dozens of switches (5–30 min per device). Knowing "5 of 20 devices done" is critical during execution.
Data migration: Scripts processing thousands of objects over many minutes.
Bulk operations: Any long-running script where incremental progress is valuable.

Database changes

None required — job.data is already a JSONField that stores script output. The log_entries ArrayField (from #19816) could also be leveraged.

External dependencies

None.

*Originally created by @berlikm on 3/24/2026* ### NetBox version v4.5.3 ### Feature type Change to existing functionality ### Proposed functionality ## NetBox version 4.5.x ## Feature type Enhancement ## Proposed functionality Currently, the script result page (`extras/htmx/script_result.html`) only renders log output after the job completes: ```html {% if job.completed %} {# Render log entries, output, tests #} {% elif job.started %} {% include 'extras/inc/result_pending.html' %} {% endif %} ``` This means users see only a spinner and **"Results pending..."** during the entire execution, even though `self.log_info()` calls accumulate data in `job.data` throughout the run. For long-running scripts (e.g., firmware upgrades across dozens of network devices taking 10–30+ minutes), this provides **no feedback on progress**. Users cannot tell if the script is working, stuck, or how far along it is. ## Proposed change Modify the `result_pending.html` include (or the conditional in `script_result.html`) to render log entries from `job.data.log` while the job is running. The HTMX auto-refresh (`every 5s`) is already in place, so the UI would naturally update as new entries appear. A minimal template change could be: ```html {% elif job.started %} {% include 'extras/inc/result_pending.html' %} {% if job.data.log %} <div class="mt-3"> <table class="table table-hover"> {% for entry in job.data.log %} <tr> <td><small class="text-muted">{{ entry.time|isodatetime }}</small></td> <td>{% badge entry.status %}</td> <td>{{ entry.message|markdown }}</td> </tr> {% endfor %} </table> </div> {% endif %} {% endif %} ``` However, **this alone is insufficient** because `job.data` is only written to the database when the script completes (in the `ScriptJob.run_script()` finally block). The data needs to be committed during execution for the HTMX refresh to pick it up. ### Why intermediate saves are hard `ScriptJob.run_script()` wraps execution in `transaction.atomic()`, so any intermediate `job.save()` within the script is **invisible to the web frontend** (different DB connection, READ COMMITTED transaction isolation). Attempting to bypass the transaction with a separate database connection (e.g., raw SQL via psycopg) causes the new connection to **block on a row-level lock** held by the script's transaction, effectively freezing the script. ### Possible approaches 1. **Periodically flush `job.data`** outside the atomic block — e.g., using a `SAVEPOINT` + partial commit, or by restructuring the transaction boundaries. 2. **Use a side channel** (Django cache, Redis pub/sub via RQ, or a separate progress table not locked by the transaction) to stream progress data to the frontend. 3. **Write to `job.log_entries`** (from [#19816](https://github.com/netbox-community/netbox/issues/19816)) via a non-transactional path and render those entries in the pending state template. 4. **Move the `job.data` save into the script loop** by providing a `flush()` or `save_progress()` method on the Script class that commits data to a separate, non-transactional table or cache. ## Current workaround Script authors can write progress to a **log file on the server** (e.g., `/var/log/`) and monitor with `tail -f`. The `_job_log()` entries (via `job.log_entries`) also appear in the **Log tab** after the job completes, but not during execution. ### Use case - **Network automation:** Scripts upgrading dozens of switches (5–30 min per device). Knowing "5 of 20 devices done" is critical during execution. - **Data migration:** Scripts processing thousands of objects over many minutes. - **Bulk operations:** Any long-running script where incremental progress is valuable. ### Database changes None required — `job.data` is already a JSONField that stores script output. The `log_entries` ArrayField (from #19816) could also be leveraged. ### External dependencies None.

Sign in to join this conversation.

Branches Tags

main

21455-sql-indexes-audit

21357-register-model-actions

feature

20924-plugin-ui-components

21766-improve-test-coverage-for-sortable-table-columns

21157-export-template-public-models

21025-pre-render-config-contexts

21364-swagger

feature-ip-prefix-link

20911-dropdown-3

fix_module_substitution

21160-filterset

20911-dropdown-2

20044-elevation-stuck-lightmode

7604-filter-modifiers-v3

20660-script-load

19724-graphql

14884-script

19720-macaddress-interface-generic-relation

fix-19669-api-image-download

7604-filter-modifiers

19275-fixes-interface-bulk-edit

fix-17794-get_field_value_return_list

11507-show-aggregate-and-rir-on-api

9583-add_column_specific_search_field_to_tables

No Label complexity: medium complexity: medium complexity: medium complexity: medium netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox status: backlog status: backlog type: feature type: feature type: feature type: feature type: feature type: feature type: feature type: feature type: feature type: feature type: feature type: feature type: feature type: feature type: feature

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/netbox#92