Custom fields with types object and multiobject generate duplicate DB queries #162

New Issue

MrUnknownDE · 2026-04-05T16:22:32+02:00

MrUnknownDE commented

2026-04-05 16:22:32 +02:00

Originally created by @miaow2 on 3/12/2026

NetBox Version

v4.5.4

Python Version

3.12

Area(s) of Concern

User Interface
REST API
GraphQL API
Python ORM
Other

Details

Hi NetBox developers!
In our company, we developed several internal NetBox plugins and added numerous custom fields with types object and multiobject from core NetBox models (particularly from VirtualMachine) to our own. We noticed a significant increase in the load time of the virtual machines API endpoint when retrieving 1000 objects per page. Using the debug toolbar, we discovered that each virtual machine object makes separate database queries for every object custom field. This appears to be an N+1 problem.
If a VirtualMachine model has 10 object custom fields, each VM object will generate an additional 10 DB queries.

How to reproduce it:

create 10 object custom fields to any model for VirtualMachine model (in my test, I added custom fields to Prefix, ClusterType, Device, Tenant, DeviceRole, Interface, Location, Platform, Rack, Region, Role models)
fill those custom fields with data
run NetBox with debug toolbar enabled
make GET request to api/virtualization/virtual-machines/
check SQL in debug tool bar

Screenshots of GET request to api/virtualization/virtual-machines/ with duplicate DB queries. I uploaded only the first several screenshots of the page. In all test requests, I use the default page size of 50 objects.

There are 340 queries in 387.51ms
If you request without custom fields api/virtualization/virtual-machines/?omit=custom_fields, then there will be 16 queries in 50.21ms. This is a very big difference.

I resolved this issue by defining ListSerializer for CustomFieldModelSerializer, which collects all primary keys from custom fields and retrieves them from the database in a single query per model, then caches the results. This approach significantly improves performance by reducing the number of database queries.

With my fix, I have only 43 queries in 70.82ms

I would be happy to submit a PR if you accept the issue. I welcome your review, as I am not certain that this is the optimal approach to fixing the problem.

*Originally created by @miaow2 on 3/12/2026* ### NetBox Version v4.5.4 ### Python Version 3.12 ### Area(s) of Concern - [ ] User Interface - [x] REST API - [ ] GraphQL API - [ ] Python ORM - [ ] Other ### Details Hi NetBox developers! In our company, we developed several internal NetBox plugins and added numerous custom fields with types `object` and `multiobject` from core NetBox models (particularly from VirtualMachine) to our own. We noticed a significant increase in the load time of the virtual machines API endpoint when retrieving 1000 objects per page. Using the debug toolbar, we discovered that each virtual machine object makes separate database queries for every object custom field. This appears to be an N+1 problem. If a VirtualMachine model has 10 object custom fields, each VM object will generate an additional 10 DB queries. How to reproduce it: 1. create 10 object custom fields to any model for VirtualMachine model (in my test, I added custom fields to Prefix, ClusterType, Device, Tenant, DeviceRole, Interface, Location, Platform, Rack, Region, Role models) 2. fill those custom fields with data 3. run NetBox with debug toolbar enabled 4. make GET request to `api/virtualization/virtual-machines/` 5. check SQL in debug tool bar Screenshots of GET request to `api/virtualization/virtual-machines/` with duplicate DB queries. I uploaded only the first several screenshots of the page. In all test requests, I use the default page size of 50 objects. <img width="2718" height="1569" alt="Image" src="https://github.com/user-attachments/assets/2c2a9d72-b05a-491e-8513-848b5b1a31f6" /> <img width="2717" height="1499" alt="Image" src="https://github.com/user-attachments/assets/0ec509b3-1890-47a3-b0b0-9a2e776149be" /> <img width="2706" height="1536" alt="Image" src="https://github.com/user-attachments/assets/bd4334cc-7fc7-47d7-a263-595cf8a78cc0" /> <img width="2709" height="1509" alt="Image" src="https://github.com/user-attachments/assets/1f01b754-61bb-4f06-a41f-cab7175893a5" /> There are 340 queries in 387.51ms If you request without custom fields `api/virtualization/virtual-machines/?omit=custom_fields`, then there will be 16 queries in 50.21ms. This is a very big difference. <img width="2723" height="1265" alt="Image" src="https://github.com/user-attachments/assets/66ac0fb8-6534-4006-872b-d96d6a348d6c" /> I resolved this issue by defining `ListSerializer` for [CustomFieldModelSerializer](https://github.com/netbox-community/netbox/blob/main/netbox/netbox/api/serializers/features.py#L17C7-L17C33), which collects all primary keys from custom fields and retrieves them from the database in a single query per model, then caches the results. This approach significantly improves performance by reducing the number of database queries. With my fix, I have only 43 queries in 70.82ms <img width="2709" height="1517" alt="Image" src="https://github.com/user-attachments/assets/a7c0d5aa-c4b8-49c9-b76e-b4e2256897f3" /> <img width="2709" height="1295" alt="Image" src="https://github.com/user-attachments/assets/514c7c00-972f-4d5e-8f1b-423187ee1721" /> I would be happy to submit a PR if you accept the issue. I welcome your review, as I am not certain that this is the optimal approach to fixing the problem.

MrUnknownDE closed this issue

2026-04-05 16:22:32 +02:00

Sign in to join this conversation.

Branches Tags

main

21455-sql-indexes-audit

21357-register-model-actions

feature

20924-plugin-ui-components

21766-improve-test-coverage-for-sortable-table-columns

21157-export-template-public-models

21025-pre-render-config-contexts

21364-swagger

feature-ip-prefix-link

20911-dropdown-3

fix_module_substitution

21160-filterset

20911-dropdown-2

20044-elevation-stuck-lightmode

7604-filter-modifiers-v3

20660-script-load

19724-graphql

14884-script

19720-macaddress-interface-generic-relation

fix-19669-api-image-download

7604-filter-modifiers

19275-fixes-interface-bulk-edit

fix-17794-get_field_value_return_list

11507-show-aggregate-and-rir-on-api

9583-add_column_specific_search_field_to_tables

No Label netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox netbox status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted status: accepted type: performance type: performance type: performance

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/netbox#162

Custom fields with types object and multiobject generate duplicate DB queries #162

NetBox Version

Python Version

Area(s) of Concern

Details

Custom fields with types `object` and `multiobject` generate duplicate DB queries #162