Compare commits

..

66 Commits

Author SHA1 Message Date
gyptazy
360920433a fix 2025-03-02 16:55:44 +01:00
gyptazy
4e39c4cba2 fix 2025-03-02 16:54:00 +01:00
gyptazy
baaf01e39b fix 2025-03-02 16:51:37 +01:00
gyptazy
0acc822777 fix 2025-03-02 16:49:46 +01:00
gyptazy
53723a0fdd fix 2025-03-02 15:51:15 +01:00
gyptazy
f451afd6d7 fix 2025-03-02 15:49:19 +01:00
gyptazy
e2aeeffdde fiux 2025-03-02 15:45:38 +01:00
gyptazy
3d33d5db44 fix 2025-03-02 15:44:28 +01:00
gyptazy
12be935115 fix 2025-03-02 15:33:58 +01:00
gyptazy
038d1e0fc3 fix 2025-03-02 15:32:26 +01:00
gyptazy
b3dd4c7e3a fix 2025-03-02 15:30:26 +01:00
gyptazy
8de8a263db fix 2025-03-02 15:26:12 +01:00
gyptazy
cf357445c6 fix 2025-03-02 15:17:05 +01:00
gyptazy
9d34d5bcd5 fix 2025-03-02 15:13:41 +01:00
gyptazy
4a3f10ccc2 fix 2025-03-02 15:09:08 +01:00
gyptazy
968d49adb0 fifx 2025-03-02 15:05:43 +01:00
gyptazy
585f6d511b fiux 2025-03-02 15:02:55 +01:00
gyptazy
c0558dda3b fix 2025-03-02 14:59:18 +01:00
gyptazy
a8470aca52 fix 2025-03-02 14:57:04 +01:00
gyptazy
d5f1853d15 fix 2025-03-02 14:51:40 +01:00
gyptazy
3da8f0638e fix 2025-03-02 14:49:45 +01:00
gyptazy
513e74c2e2 fix 2025-03-02 13:22:31 +01:00
gyptazy
3841cd80ee fix 2025-03-02 13:18:33 +01:00
gyptazy
86c9134b10 fix 2025-03-02 12:19:23 +01:00
gyptazy
ae2e15c2ae fix 2025-03-02 12:06:08 +01:00
gyptazy
1230aa2ef4 fix 2025-03-02 11:36:05 +01:00
gyptazy
f00edb6387 fix 2025-03-02 11:26:26 +01:00
gyptazy
4402a8775a fix 2025-03-02 11:25:02 +01:00
gyptazy
8dc538eb1e fix 2025-03-02 11:24:32 +01:00
gyptazy
26add60e70 fix 2025-03-02 11:22:34 +01:00
gyptazy
5b6897d250 fix 2025-03-02 11:19:05 +01:00
gyptazy
df1b9cb179 fix 2025-03-02 11:13:53 +01:00
gyptazy
57282a0273 fix 2025-03-02 11:12:51 +01:00
gyptazy
e0ed3773af fix 2025-03-01 18:33:16 +01:00
gyptazy
fb50afe473 fix 2025-03-01 18:24:18 +01:00
gyptazy
5dc747eb31 fix 2025-03-01 18:15:10 +01:00
gyptazy
87b82cfccb fix 2025-03-01 13:32:53 +01:00
gyptazy
c73bbd1538 fix 2025-03-01 13:26:57 +01:00
gyptazy
19e2d9e3ae fix 2025-03-01 13:23:35 +01:00
gyptazy
bb990f024c fix 2025-03-01 13:22:01 +01:00
gyptazy
9413d46bc4 fix 2025-03-01 13:16:18 +01:00
gyptazy
b48fd07f83 fix 2025-03-01 13:09:26 +01:00
gyptazy
a08f088bbd fix 2025-03-01 12:52:39 +01:00
gyptazy
31c455c835 fix 2025-03-01 12:49:38 +01:00
gyptazy
82704ce887 fix 2025-03-01 12:48:25 +01:00
gyptazy
8a08dd5a51 fix 2025-03-01 12:47:05 +01:00
gyptazy
389950c150 fix 2025-03-01 12:44:44 +01:00
gyptazy
8d004efc03 fix 2025-03-01 12:42:02 +01:00
gyptazy
3b5f6da2a5 fix 2025-03-01 12:40:50 +01:00
gyptazy
2f7f83f68e fix 2025-03-01 12:38:11 +01:00
gyptazy
8d46ebf758 fix 2025-03-01 12:31:48 +01:00
gyptazy
dc499c3fb6 replace ubntu pkg with pip 2025-03-01 12:23:51 +01:00
gyptazy
17da45e0aa test stdeb package ubuntu 2025-03-01 12:21:04 +01:00
gyptazy
680124a23e Fix package building 2025-03-01 12:17:25 +01:00
gyptazy
b12cb96b29 fix 2025-03-01 12:15:06 +01:00
gyptazy
66b7e58bdc use sudo 2025-03-01 12:13:35 +01:00
gyptazy
e303bddf2e add pipeline jobs 2025-03-01 12:11:38 +01:00
gyptazy
7c0fd9d76a fix 2025-03-01 10:15:05 +01:00
gyptazy
5aa1e8ee04 fix 2025-03-01 10:12:31 +01:00
gyptazy
110afb3c5f fix 2025-03-01 10:05:50 +01:00
gyptazy
9eee27afa4 fix 2025-03-01 10:03:00 +01:00
gyptazy
b3f9a2b04b fix 2025-03-01 09:58:40 +01:00
gyptazy
6c38f9ea07 fix 2025-03-01 09:56:19 +01:00
gyptazy
4900cfb53b Refactor of code base for ProxLB
* Native Python3 project
 * Oop style
 * Native Debian packaging
   - Renamed package: python3-proxlb

Fixes: #114
2025-03-01 09:47:29 +01:00
Florian Paul Azim Hoberg (@gyptazy)
e2a33e9805 fix
fix
2025-02-28 11:52:23 +01:00
gyptazy
1caf628e96 refactor: Refactor of ProxLB code base.
Fixes: #114
2025-02-28 09:42:55 +01:00
104 changed files with 325 additions and 3649 deletions

View File

@@ -1,5 +1,6 @@
fixed:
- Refactored code base for ProxLB [#114]
- Renamed package from `proxlb` to `python3-proxlb` to align with Debian packaging guidelines [#114]
- Switched to `pycodestyle` for linting [#114]
- Package building will be done within GitHub actions pipeline [#114]
- ProxLB now only returns a warning when no guests for further balancing are not present (instead of quitting) [132#]
@@ -8,4 +9,4 @@ fixed:
- Stop balancing when movement would get worste (new force param to enfoce for affinity rules) [#128]
- Added requested documentation regarding Proxmox HA groups [#127]
- Rewrite of the whole affinity/anti-affinity rules evaluation and placement [#123]
- Fixed the `ignore` parameter for nodes where the node and guests on the node will be untouched [#102]
- Fixed the `ignore` parameter for nodes where the node and guests on the node will be untouched [#102]

View File

@@ -1,2 +0,0 @@
feature:
- Add Proxmox API authentication support. [#125]

View File

@@ -1,2 +0,0 @@
fixed:
- Fix the systemd unit file to start ProxLB after pveproxy (by @robertdahlem). [#137]

View File

@@ -1 +1 @@
date: 2025-04-01
date: TBD

View File

@@ -1,2 +0,0 @@
fixed:
- Fix tag evluation for VMs for being ignored for further balancing [#163]

View File

@@ -1,2 +0,0 @@
fixed:
- Improve logging verbosity of messages that had a wrong servity [#165]

View File

@@ -1,2 +0,0 @@
feature:
- Add a more flexible way to define schedules in minutes or hours (by @gyptazy) [#168]

View File

@@ -1,2 +0,0 @@
fixed:
- Fix Python path for Docker entrypoint (by @crandler) [#170]

View File

@@ -1,2 +0,0 @@
fixed:
- Honor the value when balancing should not be performed and stop balancing [#174]

View File

@@ -1,2 +0,0 @@
changed:
- Change the default behaviour of the daemon mode to active [#176]

View File

@@ -1,2 +0,0 @@
changed:
- Change the default banalcing mode to used instead of assigned [#180]

View File

@@ -1,2 +0,0 @@
feature:
- Add validation for the minimum required permissions of a user in Proxmox [#184]

View File

@@ -1,2 +0,0 @@
fix:
- add handler to log messages with severity less than info to the screen when there is no systemd integration, for instance, inside a docker container (by @glitchvern) [#185]

View File

@@ -1,2 +0,0 @@
fixed:
- allow the use of minutes instead of hours and only accept hours or minutes in the format (by @glitchvern) [#187]

View File

@@ -1,2 +0,0 @@
fixed:
- Set cpu_used to the cpu usage, which is a percent, times the total number of cores to get a number where guest cpu_used can be added to nodes cpu_used and be meaningful (by @glitchvern) [#195]

View File

@@ -1,2 +0,0 @@
fixed:
- Remove hard coded memory usage from lowest usage node and use method and mode specified in configuration instead (by @glitchvern) [#197]

View File

@@ -1,2 +0,0 @@
fixed:
- Requery a guest if that running guest reports 0 cpu usage (by @glitchvern) [#200]

View File

@@ -1,2 +0,0 @@
fixed:
- Fix the guest type relationship in the logs when a migration job failed (by @gyptazy) [#204]

View File

@@ -1,2 +0,0 @@
added:
- Providing the API upstream error message when migration fails in debug mode (by @gyptazy) [#205]

View File

@@ -1 +0,0 @@
date: 2025-04-20

View File

@@ -1,2 +0,0 @@
added:
- Prevent redundant rebalancing by validating existing affinity enforcement before taking actions (@gyptazy). [#335]

View File

@@ -1,2 +0,0 @@
added:
- Add safety-guard for PVE 8 users when activating conntrack-aware migrations mistakenly (@gyptazy). [#359]

View File

@@ -1,3 +0,0 @@
fixed:
- Fixed the Proxmox API connection validation which returned a false-positive logging message of timeouts (@gyptazy). [#361]
- Refactored Proxmox API connection functions

View File

@@ -1,2 +0,0 @@
fixed:
- Fixed a crash during PVE resource pool enumeration by skipping members not having a 'name' property (@stefanoettl). [#368]

View File

@@ -1 +0,0 @@
date: 2025-11-25

View File

@@ -1,2 +0,0 @@
fixed:
- Fix systemd unit file to run after network on non PVE nodes (by @robertdahlem) [#137]

View File

@@ -1,2 +0,0 @@
added:
- Add a configurable retry mechanism when connecting to the Proxmox API (by @gyptazy) [#157]

View File

@@ -1,2 +0,0 @@
added:
- Add 1-to-1 relationships between guest and hypervisor node to ping a guest on a node (by @gyptazy) [#218]

View File

@@ -1,2 +0,0 @@
fixed:
- Force type cast cpu count of guests to int for some corner cases where a str got returned (by @gyptazy). [#222]

View File

@@ -1 +0,0 @@
date: 2025-05-13

View File

@@ -1,2 +0,0 @@
added:
- Add relaod (SIGHUP) function to ProxLB to reload the configuration (by @gyptazy). [#189]

View File

@@ -1,2 +0,0 @@
fixed:
- Align maintenance mode with Proxmox HA maintenance mode (by @gyptazy). [#232]

View File

@@ -1,2 +0,0 @@
added:
- Add optional wait time parameter to delay execution until the service takes action (by @gyptazy). #239

View File

@@ -1,2 +0,0 @@
added:
- Make the amount of parallel migrations configurable (by @gyptazy). [#241]

View File

@@ -1,2 +0,0 @@
changed:
- Use the average CPU consumption of a guest within the last 60 minutes instead of the current CPU usage (by @philslab-ninja & @gyptazy). [#94]

View File

@@ -1 +0,0 @@
date: 2025-06-19

View File

@@ -1,2 +0,0 @@
added:
- Allow pinning of guests to a group of nodes (@gyptazy). [#245]

View File

@@ -1,2 +0,0 @@
fixed:
- Fixed an issue where balancing was performed in combination of deactivated balancing and dry-run mode (@gyptazy). [#248]

View File

@@ -1,2 +0,0 @@
fixed:
- Modified log levels to make output lighter at INFO level (@pmarasse) [#255]

View File

@@ -1 +0,0 @@
date: 2025-06-27

View File

@@ -1,2 +0,0 @@
added:
- Allow custom API ports instead of fixed tcp/8006 (@gyptazy). [#260]

View File

@@ -1 +0,0 @@
date: 2025-07-14

View File

@@ -1,2 +0,0 @@
fixed:
- Fix balancing evaluation of guest types (e.g., VM or CT) (@gyptazy). [#268]

View File

@@ -1,2 +0,0 @@
added:
- Add validation for provided API user token id to avoid confusions (@gyptazy). [#291]

View File

@@ -1,2 +0,0 @@
fixed:
- Fix stacktrace output when validating permissions on non existing users in Proxmox (@gyptazy). [#291]

View File

@@ -1,3 +0,0 @@
fixed:
- Fix Overprovisioning first node if anti_affinity_group has only one member (@MiBUl-eu). [#295]

View File

@@ -1,3 +0,0 @@
fixed:
- Validate for node presence when pinning guests to avoid crashing (@gyptazy). [#296]

View File

@@ -1 +0,0 @@
date: 2025-09-04

View File

@@ -1,2 +0,0 @@
added:
- Add graceful shutdown for SIGINT (e.g., CTRL + C abort). (@gyptazy). [#304]

View File

@@ -1,2 +0,0 @@
added:
- Add conntrack state aware migrations of VMs (@gyptazy). [#305]

View File

@@ -1,2 +0,0 @@
fixed:
- Fix crash when validating absent migration job ids. (@gyptazy). [#308]

View File

@@ -1,2 +0,0 @@
fixed:
- Fix guest object names are not being evaluated in debug log. (@gyptazy). [#310]

View File

@@ -1 +0,0 @@
date: 2025-09-19

View File

@@ -1,3 +0,0 @@
changed:
- Container image does not run as root anymore (@mikaelkrantz945). [#317]
- Container image uses venv for running ProxLB (@mikaelkrantz945). [#317]

View File

@@ -1,2 +0,0 @@
fixed:
- Fix API errors when using conntrack aware migration with older PVE versions (@gyptazy). [#318]

View File

@@ -1,2 +0,0 @@
fixed:
- Add a static ProxLB prefix to the log output when used by journal handler (@gyptazy). [#329]

View File

@@ -1 +0,0 @@
date: 2025-10-09

View File

@@ -1,5 +0,0 @@
added:
- Add pressure (PSI) based balancing for memory, cpu, disk (req. PVE9 or greater) (@gyptazy). [#337]
- Pressure (PSI) based balancing for nodes
- Pressure (PSI) based balancing for guests
- Add PVE version evaluation

View File

@@ -1,2 +0,0 @@
added:
- Add an optional memory balancing threshold (@gyptazy). [#342]

View File

@@ -1,2 +0,0 @@
added:
- Add affinity/anti-affinity support by pools (@gyptazy). [#343]

View File

@@ -1 +0,0 @@
date: 2025-10-30

View File

@@ -28,7 +28,7 @@ jobs:
- name: Check out repository
uses: actions/checkout@v3
with:
ref: ${{ github.ref }}
ref: 'development'
- name: Set up Docker with Debian image
run: |
@@ -39,13 +39,9 @@ jobs:
docker run --rm -v $(pwd):/workspace -w /workspace debian:latest bash -c "
# Install dependencies
apt-get update && \
apt-get install -y python3 python3-setuptools debhelper dh-python python3-pip python3-stdeb python3-proxmoxer python3-requests python3-urllib3 devscripts python3-all && \
# Build package using stdeb / setuptools
# python3 setup.py --command-packages=stdeb.command bdist_deb && \
# Build native package
dpkg-buildpackage -us -uc && \
mkdir package && \
mv ../*.deb package/ && \
apt-get install -y python3 python3-setuptools debhelper dh-python python3-pip python3-stdeb python3-proxmoxer python3-requests python3-urllib3 && \
# Build package
python3 setup.py --command-packages=stdeb.command bdist_deb && \
echo 'OK: Debian package successfully created.'
"
@@ -53,35 +49,25 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: debian-package
path: package/*.deb
path: deb_dist/*.deb
integration-test-debian:
needs: build-package-debian
runs-on: ubuntu-latest
strategy:
matrix:
debian_version: [bookworm, trixie]
name: Integration Test on Debian ${{ matrix.debian_version }}
steps:
- name: Download Debian package artifact
uses: actions/download-artifact@v4
with:
name: debian-package
path: package/
path: deb_dist/
- name: Set up Docker with Debian image
run: docker pull debian:${{ matrix.debian_version }}
run: docker pull debian:latest
- name: Install and test Debian package in Docker container
run: |
docker run --rm \
-v "$(pwd)/package:/package" \
-w /package \
debian:${{ matrix.debian_version }} \
bash -c "
set -e
apt-get update
apt-get install -y python3 systemd
apt-get install -y ./proxlb*.deb
python3 -c 'import proxlb; print(\"OK: Debian package successfully installed on ${{ matrix.debian_version }}.\")'
"
docker run --rm -v $(pwd)/deb_dist:/deb_dist -w /deb_dist debian:latest bash -c "
apt-get update && \
apt-get install -y ./python3-proxlb*.deb && \
python3 -c 'import proxlb; print(\"OK: Debian package successfully installed.\")'
"

View File

@@ -1,26 +0,0 @@
name: "Build Container Image: AMD64"
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build amd64 image and save as tar
run: |
docker buildx build \
--platform linux/amd64 \
--load \
-t proxlb-image:amd64 \
.
docker save proxlb-image:amd64 -o proxlb_image_amd64.tar
- name: Upload Docker image artifact
uses: actions/upload-artifact@v4
with:
name: proxlb-image-amd64
path: proxlb_image_amd64.tar

View File

@@ -1,26 +0,0 @@
name: "Build Container Image: ARM64"
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build arm64 image and save as tar
run: |
docker buildx build \
--platform linux/arm64 \
--load \
-t proxlb-image:arm64 \
.
docker save proxlb-image:arm64 -o proxlb_image_arm64.tar
- name: Upload Docker image artifact
uses: actions/upload-artifact@v4
with:
name: proxlb-image-arm64
path: proxlb_image_arm64.tar

View File

@@ -1,23 +0,0 @@
name: "Build Container Image: Multiarch"
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build multi-arch image and save as tar
run: |
docker buildx build \
--platform linux/amd64,linux/arm64 \
--output type=tar,dest=proxlb_image_multiarch.tar \
.
- name: Upload Docker image artifact
uses: actions/upload-artifact@v4
with:
name: proxlb-image-multiarch
path: proxlb_image_multiarch.tar

View File

@@ -1,284 +0,0 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.1.10] - 2025-11-25
### Added
- Prevent redundant rebalancing by validating existing affinity enforcement before taking actions (@gyptazy). [#335]
- Add safety-guard for PVE 8 users when activating conntrack-aware migrations mistakenly (@gyptazy). [#359]
### Fixed
- Fix the Proxmox API connection validation which returned a false-positive logging message of timeouts (@gyptazy). [#361]
- Refactored Proxmox API connection functions (@gyptazy). [#361]
- Fix a crash during PVE resource pool enumeration by skipping members not having a 'name' property (@stefanoettl). [#368]
## [1.1.9.1] - 2025-10-30
### Fixed
- Fix quoting in f-strings which may cause issues on PVE 8 / Debian Bookworm systems (@gyptazy). [#352]
## [1.1.9] - 2025-10-30
### Added
- Add an optional memory balancing threshold (@gyptazy). [#342]
- Add affinity/anti-affinity support by pools (@gyptazy). [#343]
- Add pressure (PSI) based balancing for memory, cpu, disk (req. PVE9 or greater) (@gyptazy). [#337]
- Pressure (PSI) based balancing for nodes
- Pressure (PSI) based balancing for guests
- Add PVE version evaluation
## [1.1.8] - 2025-10-09
### Fixed
- Fix API errors when using conntrack aware migration with older PVE versions (@gyptazy). [#318]
- Add a static ProxLB prefix to the log output when used by journal handler (@gyptazy). [#329]
### Changed
- Container image does not run as root anymore (@mikaelkrantz945). [#317]
- Container image uses venv for running ProxLB (@mikaelkrantz945). [#317]
## [1.1.7] - 2025-09-19
### Added
- Add conntrack state aware migrations of VMs (@gyptazy). [#305]
- Add graceful shutdown for SIGINT (e.g., CTRL + C abort). (@gyptazy). [#304]
### Fixed
- Fix crash when validating absent migration job ids. (@gyptazy). [#308]
- Fix guest object names are not being evaluated in debug log. (@gyptazy). [#310]
## [1.1.6.1] - 2025-09-04
### Fixed
- Validate for node presence when pinning VMs to avoid crashing (@gyptazy). [#296]
## [1.1.6] - 2025-09-04
### Added
- Add validation for provided API user token id to avoid confusions (@gyptazy). [#291]
### Fixed
- Fix stacktrace output when validating permissions on non existing users in Proxmox (@gyptazy). [#291]
- Fix Overprovisioning first node if anti_affinity_group has only one member (@MiBUl-eu). [#295]
- Validate for node presence when pinning guests to avoid crashing (@gyptazy). [#296]
- Fix balancing evaluation of guest types (e.g., VM or CT) (@gyptazy). [#268]
## [1.1.5] - 2025-07-14
### Added
- Allow custom API ports instead of fixed tcp/8006 (@gyptazy). [#260]
## [1.1.4] - 2025-06-27
### Added
- Allow pinning of guests to a group of nodes (@gyptazy). [#245]
### Fixed
- Modified log levels to make output lighter at INFO level (@pmarasse) [#255]
- Fixed an issue where balancing was performed in combination of deactivated balancing and dry-run mode (@gyptazy). [#248]
## [1.1.3] - 2025-06-19
### Added
- Add relaod (SIGHUP) function to ProxLB to reload the configuration (by @gyptazy). [#189]
- Add optional wait time parameter to delay execution until the service takes action (by @gyptazy). [#239]
- Make the amount of parallel migrations configurable (by @gyptazy). [#241]
### Changed
- Use the average CPU consumption of a guest within the last 60 minutes instead of the current CPU usage (by @philslab-ninja & @gyptazy). [#94]
### Fixed
- Align maintenance mode with Proxmox HA maintenance mode (by @gyptazy). [#232]
## [1.1.2] - 2025-05-13
### Added
- Add a configurable retry mechanism when connecting to the Proxmox API (by @gyptazy) [#157]
- Add 1-to-1 relationships between guest and hypervisor node to ping a guest on a node (by @gyptazy) [#218]
### Fixed
- Force type cast cpu count of guests to int for some corner cases where a str got returned (by @gyptazy). [#222]
- Fix systemd unit file to run after network on non PVE nodes (by @robertdahlem) [#137]
## [1.1.1] - 2025-04-20
### Added
- Providing the API upstream error message when migration fails in debug mode (by @gyptazy) [#205]
### Changed
- Change the default behaviour of the daemon mode to active [#176]
- Change the default banalcing mode to used instead of assigned [#180]
### Fixed
- Set cpu_used to the cpu usage, which is a percent, times the total number of cores to get a number where guest cpu_used can be added to nodes cpu_used and be meaningful (by @glitchvern) [#195]
- Fix tag evluation for VMs for being ignored for further balancing [#163]
- Honor the value when balancing should not be performed and stop balancing [#174]
- allow the use of minutes instead of hours and only accept hours or minutes in the format (by @glitchvern) [#187]
- Remove hard coded memory usage from lowest usage node and use method and mode specified in configuration instead (by @glitchvern) [#197]
- Fix the guest type relationship in the logs when a migration job failed (by @gyptazy) [#204]
- Requery a guest if that running guest reports 0 cpu usage (by @glitchvern) [#200]
- Fix Python path for Docker entrypoint (by @crandler) [#170]
- Improve logging verbosity of messages that had a wrong servity [#165]
## [1.1.0] - 2025-04-01
### Fixed
- Refactored code base for ProxLB [#114]
- Switched to `pycodestyle` for linting [#114]
- Package building will be done within GitHub actions pipeline [#114]
- ProxLB now only returns a warning when no guests for further balancing are not present (instead of quitting) [132#]
- All nodes (according to the free resources) will be used now [#130]
- Fixed logging outputs where highest/lowest were mixed-up [#129]
- Stop balancing when movement would get worste (new force param to enfoce for affinity rules) [#128]
- Added requested documentation regarding Proxmox HA groups [#127]
- Rewrite of the whole affinity/anti-affinity rules evaluation and placement [#123]
- Fixed the `ignore` parameter for nodes where the node and guests on the node will be untouched [#102]
## [1.0.6] - 2024-12-24
### Fixed
- Fix maintenance mode when using cli arg and config mode by using the merged list (by @CartCaved). [#119]
- Fix that a scheduler time definition of 1 (int) gets wrongly interpreted as a bool (by @gyptazy). [#115]
## [1.0.5] - 2024-10-30
### Changed
- Change docs to make bool usage in configs more clear (by @gyptazy). [#104]
### Fixed
- Fix node (and its objects) evaluation when not reachable, e.g., maintenance (by @gyptazy). [#107]
- Fix migration from local disks (by @greenlogles). [#113]
- Fix evaluation of maintenance mode where comparing list & string resulted in a crash (by @glitchvern). [#106]
- Fix allowed values (add DEBUG, WARNING) for log verbosity (by @gyptazy). [#98]
## [1.0.4] - 2024-10-11
### Added
- Add maintenance mode to evacuate a node and move workloads for other nodes in the cluster. [#58]
- Add feature to make API timeout configureable. [#91]
- Add version output cli arg. [#89]
### Changed
- Run storage balancing only on supported shared storages. [#79]
- Run storage balancing only when needed to save time. [#79]
### Fixed
- Fix CPU balancing where calculations are done in float instead of int. (by @glitchvern) [#75]
- Fix documentation for the underlying infrastructure. [#81]
## [1.0.3] - 2024-09-12
### Added
- Add cli arg `-b` to return the next best node for next VM/CT placement. [#8]
- Add a convert function to cast all bool alike options from configparser to bools. [#53]
- Add a config parser options for future features. [#53]
- Add a config versio schema that must be supported by ProxLB. [#53]
- Add feature to allow the API hosts being provided as a comma separated list. [#60]
- Add doc how to add dedicated user for authentication. (by @Dulux-Oz)
- Add storage balancing function. [#51]
### Changed
- Provide a more reasonable output when HA services are not active in a Proxmox cluster. [#68]
- Improve the underlying code base for future implementations. [#53]
### Fixed
- Fix anti-affinity rules not evaluating a new and different node. [#67]
- Fixed `master_only` function by inverting the condition.
- Fix documentation for the master_only parameter placed in the wrong config section. [#74]
- Fix bug in the `proxlb.conf` in the vm_balancing section.
- Fix handling of unset `ignore_nodes` and `ignore_vms` resulted in an attribute error. [#71]
- Improved the overall validation and error handling. [#64]
## [1.0.2] - 2024-08-13
### Added
- Add option to run ProxLB only on the Proxmox's master node in the cluster (reg. HA feature). [#40]
- Add option to run migrations in parallel or sequentially. [#41]
### Changed
- Fix daemon timer to use hours instead of minutes. [#45]
### Fixed
- Fix CMake packaging for Debian package to avoid overwriting the config file. [#49]
## [1.0.0] - 2024-08-01
### Added
- Add feature to prevent VMs from being relocated by defining the 'plb_ignore_vm' tag. [#7]
- Add feature to prevent VMs from being relocated by defining a wildcard pattern. [#7]
- Add Docker/Podman support. [#10 by @daanbosch]
- Add option to rebalance by assigned VM resources to avoid overprovisioning. [#16]
- Add feature to make log verbosity configurable [#17].
- Add dry-run support to see what kind of rebalancing would be done. [#6]
- Add LXC/Container integration. [#27]
- Add exclude grouping feature to rebalance VMs from being located together to new nodes. [#4]
- Add include grouping feature to rebalance VMs bundled to new nodes. [#3]
- Add option_mode to rebalance by node's free resources in percent (instead of bytes). [#29]
### Changed
- Adjusted general logging and log more details.
## [0.9.9] - 2024-07-06
### Added
- Initial public development release of ProxLB.
## [0.9.0] - 2024-02-01
### Added
- Development release of ProxLB.

View File

@@ -1,41 +0,0 @@
# Use the latest Alpine image
FROM alpine:latest
# Labels
LABEL maintainer="gyptazy@gyptazy.com"
LABEL org.label-schema.name="ProxLB"
LABEL org.label-schema.description="ProxLB - An advanced load balancer for Proxmox clusters."
LABEL org.label-schema.vendor="gyptazy"
LABEL org.label-schema.url="https://proxlb.de"
LABEL org.label-schema.vcs-url="https://github.com/gyptazy/ProxLB"
# --- Step 1 (root): system deps, user, dirs ---
RUN apk add --no-cache python3 py3-pip \
&& addgroup -S plb \
&& adduser -S -G plb -h /home/plb plb \
&& mkdir -p /app/conf /opt/venv \
&& chown -R plb:plb /app /home/plb /opt/venv
WORKDIR /app
# Copy only requirements first for better layer caching
COPY --chown=plb:plb requirements.txt /app/requirements.txt
# --- Step 2 (appuser): venv + deps + code ---
USER plb
# Create venv owned by appuser and put it on PATH
RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:${PATH}"
# Install Python dependencies into the venv (no PEP 668 issues)
RUN pip install --no-cache-dir -r /app/requirements.txt
# Copy application code (owned by appuser)
COPY --chown=plb:plb proxlb /app/proxlb
# Optional: placeholder config so a bind-mount can override cleanly
RUN touch /app/conf/proxlb.yaml
# Run as non-root using venv Python
ENTRYPOINT ["/opt/venv/bin/python", "/app/proxlb/main.py"]

416
README.md
View File

@@ -1,19 +1,26 @@
# ProxLB - (Re)Balance VM Workloads in Proxmox Clusters
<img align="left" src="https://cdn.gyptazy.com/img/ProxLB.jpg"/>
<img align="left" src="https://cdn.gyptazy.com/images/Prox-LB-logo.jpg"/>
<br>
<p float="center"><img src="https://img.shields.io/github/license/gyptazy/ProxLB"/><img src="https://img.shields.io/github/contributors/gyptazy/ProxLB"/><img src="https://img.shields.io/github/last-commit/gyptazy/ProxLB/main"/><img src="https://img.shields.io/github/issues-raw/gyptazy/ProxLB"/><img src="https://img.shields.io/github/issues-pr/gyptazy/ProxLB"/></p>
# :warning: Important: ProxLB 1.1.x is coming
This repository is currently under heavy work and changes. During that time it might come to issues, non working pipelines or wrong documentation. Please select a stable release tag for a suitable version during this time!
## Table of Contents
1. [Introduction](#introduction)
2. [Features](#features)
3. [How does it work?](#how-does-it-work)
4. [Documentation](#documentation)
5. [Installation](#installation)
4. [Installation](#installation)
1. [Requirements / Dependencies](#requirements--dependencies)
2. [Debian Package](#debian-package)
3. [RedHat Package](#redhat-package)
4. [Container / Docker](#container--docker)
5. [Source](#source)
5. [Upgrading](#upgrading)
1. [Upgrading from < 1.1.0](#upgrading-from--110)
2. [Upgrading from >= 1.1.0](#upgrading-from--110)
6. [Usage / Configuration](#usage--configuration)
1. [GUI Integration](#gui-integration)
2. [Proxmox HA Integration](#proxmox-ha-integration)
@@ -21,19 +28,17 @@
7. [Affinity & Anti-Affinity Rules](#affinity--anti-affinity-rules)
1. [Affinity Rules](#affinity-rules)
2. [Anti-Affinity Rules](#anti-affinity-rules)
3. [Ignore VMs](#ignore-vms)
4. [Pin VMs to Hypervisor Nodes](#pin-vms-to-hypervisor-nodes)
8. [Maintenance](#maintenance)
9. [Misc](#misc)
1. [Bugs](#bugs)
2. [Contributing](#contributing)
3. [Support](#support)
4. [Enterprise-Support](#enterprise-support)
3. [Documentation](#documentation)
4. [Support](#support)
10. [Author(s)](#authors)
## Introduction
ProxLB is an advanced load balancing solution specifically designed for Proxmox clusters, addressing the absence of an intelligent and more advanced resource scheduler. As a third-party solution, ProxLB enhances the management and efficiency of Proxmox clusters by intelligently distributing workloads across available nodes. Workloads can be balanced by different times like the guest's memory, CPU or disk usage or their assignment to avoid overprovisioning and ensuring resources.
ProxLB is an advanced load balancing solution specifically designed for Proxmox clusters, addressing the absence of a Dynamic Resource Scheduler (DRS) that is familiar to VMware users. As a third-party solution, ProxLB enhances the management and efficiency of Proxmox clusters by intelligently distributing workloads across available nodes. Workloads can be balanced by different times like the guest's memory, CPU or disk usage or their assignment to avoid overprovisioning and ensuring resources.
One of the key advantages of ProxLB is that it is fully open-source and free, making it accessible for anyone to use, modify, and contribute to. This ensures transparency and fosters community-driven improvements. ProxLB supports filtering and ignoring specific nodes and guests through configuration files and API calls, providing administrators with the flexibility to tailor the load balancing behavior to their specific needs.
@@ -44,7 +49,7 @@ ProxLB can also return the best next node for guest placement, which can be inte
Overall, ProxLB significantly enhances resource management by intelligently distributing workloads, reducing downtime through its maintenance mode, and providing improved flexibility with affinity and anti-affinity rules. Its seamless integration with CI/CD tools and reliance on the Proxmox API make it a robust and secure solution for optimizing Proxmox cluster performance.
### Video of Migration
<img src="https://cdn.gyptazy.com/img/proxlb-rebalancing-demo.gif"/>
<img src="https://cdn.gyptazy.com/images/proxlb-rebalancing-demo.gif"/>
## Features
ProxLB's key features are by enabling automatic rebalancing of VMs and CTs across a Proxmox cluster based on memory, CPU, and local disk usage while identifying optimal nodes for automation. It supports maintenance mode, affinity rules, and seamless Proxmox API integration with ACL support, offering flexible usage as a one-time operation, a daemon, or through the Proxmox Web GUI.
@@ -54,10 +59,6 @@ ProxLB's key features are by enabling automatic rebalancing of VMs and CTs acros
* Memory
* Disk (only local storage)
* CPU
* Rebalance by different modes:
* Used resources
* Assigned resources
* PSI (Pressure) of resources
* Get best nodes for further automation
* Supported Guest Types
* VMs
@@ -79,16 +80,9 @@ ProxLB is a load-balancing system designed to optimize the distribution of virtu
Before starting any migrations, ProxLB validates that rebalancing actions are necessary and beneficial. Depending on the selected balancing mode — such as CPU, memory, or disk — it creates a balancing matrix. This matrix sorts the VMs by their maximum used or assigned resources, identifying the VM with the highest usage. ProxLB then places this VM on the node with the most free resources in the selected balancing type. This process runs recursively until the operator-defined Balanciness is achieved. Balancing can be defined for the used or max. assigned resources of VMs/CTs.
## Documentation
This `README.md` doesn't contain all information and only highlights the most important facts. Extended information, such like API permissions, creating dedicated user, best-practices in running ProxLB and much more can be found in the [docs/](https://github.com/gyptazy/ProxLB/tree/main/docs) directory. Please consult the documentation before creating issues.
## Installation
### Requirements / Dependencies
* Proxmox
* Proxmox 7.x
* Proxmox 8.x
* Proxmox 9.x
* Python3.x
* proxmoxer
* requests
@@ -100,144 +94,29 @@ The dependencies can simply be installed with `pip` by running the following com
pip install -r requirements.txt
```
*Note: Distribution packages, such like the provided `.deb` package will automatically resolve and install all required dependencies by using already packaged version from the distribution's repository. By using the Docker (container) image or Debian packages, you do not need to take any care of the requirements listed here.*
Distribution packages, such like the provided `.deb` package will automatically resolve and install all required dependencies by using already packaged version from the distribution's repository.
### Debian Package
ProxLB is a powerful and flexible load balancer designed to work across various architectures, including `amd64`, `arm64`, `rv64` and many other ones that support Python. It runs independently of the underlying hardware, making it a versatile choice for different environments. This chapter covers the step-by-step process to install ProxLB on Debian-based systems, including Debian clones like Ubuntu.
#### Quick-Start
You can simply use this snippet to install the repository and to install ProxLB on your system.
### RedHat Package
```bash
echo "deb https://repo.gyptazy.com/stable /" > /etc/apt/sources.list.d/proxlb.list
wget -O /etc/apt/trusted.gpg.d/proxlb.asc https://repo.gyptazy.com/repository.gpg
apt-get update && apt-get -y install proxlb
cp /etc/proxlb/proxlb_example.yaml /etc/proxlb/proxlb.yaml
# Adjust the config to your needs
vi /etc/proxlb/proxlb.yaml
systemctl start proxlb
```
Afterwards, ProxLB is running in the background and balances your cluster by your defined balancing method (default: memory).
#### Details
ProxLB provides two different repositories:
* https://repo.gyptazy.com/stable (only stable release)
* https://repo.gyptazy.com/testing (bleeding edge - not recommended)
The repository is signed and the GPG key can be found at:
* https://repo.gyptazy.com/repository.gpg
You can also simply import it by running:
```
# KeyID: 17169F23F9F71A14AD49EDADDB51D3EB01824F4C
# UID: gyptazy Solutions Repository <contact@gyptazy.com>
# SHA256: 52c267e6f4ec799d40cdbdb29fa518533ac7942dab557fa4c217a76f90d6b0f3 repository.gpg
wget -O /etc/apt/trusted.gpg.d/proxlb.asc https://repo.gyptazy.com/repository.gpg
```
*Note: The defined repositories `repo.gyptazy.com` and `repo.proxlb.de` are the same!*
#### Debian Packages (.deb files)
If you do not want to use the repository you can also find the debian packages as a .deb file on gyptazy's CDN at:
* https://cdn.gyptazy.com/debian/proxlb/
Afterwards, you can simply install the package by running:
```bash
dpkg -i proxlb_*.deb
cp /etc/proxlb/proxlb_example.yaml /etc/proxlb/proxlb.yaml
# Adjust the config to your needs
vi /etc/proxlb/proxlb.yaml
systemctl start proxlb
```
### Container Images / Docker
Using the ProxLB container images is straight forward and only requires you to mount the config file.
```bash
# Pull the image
docker pull cr.gyptazy.com/proxlb/proxlb:latest
# Download the config
wget -O proxlb.yaml https://raw.githubusercontent.com/gyptazy/ProxLB/refs/heads/main/config/proxlb_example.yaml
# Adjust the config to your needs
vi proxlb.yaml
# Start the ProxLB container image with the ProxLB config
docker run -it --rm -v $(pwd)/proxlb.yaml:/etc/proxlb/proxlb.yaml proxlb
```
*Note: ProxLB container images are officially only available at cr.proxlb.de and cr.gyptazy.com.*
#### Overview of Images
| Version | Image |
|------|:------:|
| latest | cr.gyptazy.com/proxlb/proxlb:latest |
| v1.1.10 | cr.gyptazy.com/proxlb/proxlb:v1.1.10 |
| v1.1.9.1 | cr.gyptazy.com/proxlb/proxlb:v1.1.9.1 |
| v1.1.9 | cr.gyptazy.com/proxlb/proxlb:v1.1.9 |
| v1.1.8 | cr.gyptazy.com/proxlb/proxlb:v1.1.8 |
| v1.1.7 | cr.gyptazy.com/proxlb/proxlb:v1.1.7 |
| v1.1.6.1 | cr.gyptazy.com/proxlb/proxlb:v1.1.6.1 |
| v1.1.6 | cr.gyptazy.com/proxlb/proxlb:v1.1.6 |
| v1.1.5 | cr.gyptazy.com/proxlb/proxlb:v1.1.5 |
| v1.1.4 | cr.gyptazy.com/proxlb/proxlb:v1.1.4 |
| v1.1.3 | cr.gyptazy.com/proxlb/proxlb:v1.1.3 |
| v1.1.2 | cr.gyptazy.com/proxlb/proxlb:v1.1.2 |
| v1.1.1 | cr.gyptazy.com/proxlb/proxlb:v1.1.1 |
| v1.1.0 | cr.gyptazy.com/proxlb/proxlb:v1.1.0 |
| v1.0.6 | cr.gyptazy.com/proxlb/proxlb:v1.0.6 |
| v1.0.5 | cr.gyptazy.com/proxlb/proxlb:v1.0.5 |
| v1.0.4 | cr.gyptazy.com/proxlb/proxlb:v1.0.4 |
| v1.0.3 | cr.gyptazy.com/proxlb/proxlb:v1.0.3 |
| v1.0.2 | cr.gyptazy.com/proxlb/proxlb:v1.0.2 |
| v1.0.0 | cr.gyptazy.com/proxlb/proxlb:v1.0.0 |
| v0.9.9 | cr.gyptazy.com/proxlb/proxlb:v0.9.9 |
### Container / Docker
### Source
ProxLB can also easily be used from the provided sources - for traditional systems but also as a Docker/Podman container image.
#### Traditional System
Setting up and running ProxLB from the sources is simple and requires just a few commands. Ensure Python 3 and the Python dependencies are installed on your system, then run ProxLB using the following command:
```bash
git clone https://github.com/gyptazy/ProxLB.git
cd ProxLB
```
## Upgrading
Afterwards simply adjust the config file to your needs:
```bash
vi config/proxlb.yaml
```
### Upgrading from < 1.1.0
Upgrading ProxLB is not supported due to a fundamental redesign introduced in version 1.1.x. With this update, ProxLB transitioned from a monolithic application to a pure Python-style project, embracing a more modular and flexible architecture. This shift aimed to improve maintainability and extensibility while keeping up with modern development practices. Additionally, ProxLB moved away from traditional ini-style configuration files and adopted YAML for configuration management. This change simplifies configuration handling, reduces the need for extensive validation, and ensures better type casting, ultimately providing a more streamlined and user-friendly experience.
Start ProxLB by Python3 on the system:
```bash
python3 proxlb/main.py -c config/proxlb.yaml
```
#### Container Image
Creating a container image of ProxLB is straightforward using the provided Dockerfile. The Dockerfile simplifies the process by automating the setup and configuration required to get ProxLB running in an Alpine container. Simply follow the steps in the Dockerfile to build the image, ensuring all dependencies and configurations are correctly applied. For those looking for an even quicker setup, a ready-to-use ProxLB container image is also available, eliminating the need for manual building and allowing for immediate deployment.
```bash
git clone https://github.com/gyptazy/ProxLB.git
cd ProxLB
docker build -t proxlb .
```
Afterwards simply adjust the config file to your needs:
```bash
vi config/proxlb.yaml
```
Finally, start the created container.
```bash
docker run -it --rm -v $(pwd)/proxlb.yaml:/etc/proxlb/proxlb.yaml proxlb
```
### Upgrading from >= 1.1.0
Uprading within the current stable versions, starting from 1.1.0, will be possible in all supported ways.
## Usage / Configuration
Running ProxLB is straightforward and versatile, as it only requires `Python3` and the `proxmoxer` library. This means ProxLB can be executed directly on a Proxmox node or on dedicated systems such as Debian, RedHat, or even FreeBSD, provided that the Proxmox API is accessible from the client running ProxLB. ProxLB can also run inside a Container - Docker or LXC - and is simply up to you.
### GUI Integration
<img align="left" src="https://cdn.gyptazy.com/img/rebalance-ui.jpg"/> ProxLB can also be accessed through the Proxmox Web UI by installing the optional `pve-proxmoxlb-service-ui` package, which depends on the proxlb package. For full Web UI integration, this package must be installed on all nodes within the cluster. Once installed, a new menu item - `Rebalancing`, appears in the cluster level under the HA section. Once installed, it offers two key functionalities:
<img align="left" src="https://cdn.gyptazy.com/images/proxlb-GUI-integration.jpg"/> ProxLB can also be accessed through the Proxmox Web UI by installing the optional `pve-proxmoxlb-service-ui` package, which depends on the proxlb package. For full Web UI integration, this package must be installed on all nodes within the cluster. Once installed, a new menu item - `Rebalancing`, appears in the cluster level under the HA section. Once installed, it offers two key functionalities:
* Rebalancing VM workloads
* Migrate VM workloads away from a defined node (e.g. maintenance preparation)
@@ -257,64 +136,42 @@ See also: [#65: Host groups: Honour HA groups](https://github.com/gyptazy/ProxLB
### Options
The following options can be set in the configuration file `proxlb.yaml`:
| Section | Option | Sub Option | Example | Type | Description |
|---------|:------:|:----------:|:-------:|:----:|:-----------:|
| `proxmox_api` | | | | | |
| | hosts | | ['virt01.example.com', '10.10.10.10', 'fe01:bad:code::cafe', 'virt01.example.com:443', '[fc00::1]', '[fc00::1]:443', 'fc00::1:8006'] | `List` | List of Proxmox nodes. Can be IPv4, IPv6 or mixed. You can specify custom ports. In case of IPv6 without brackets the port is considered after the last colon |
| | user | | root@pam | `Str` | Username for the API. |
| | pass | | FooBar | `Str` | Password for the API. (Recommended: Use API token authorization!) |
| | token_id | | proxlb | `Str` | Token ID of the user for the API. |
| | token_secret | | 430e308f-1337-1337-beef-1337beefcafe | `Str` | Secret of the token ID for the API. |
| | ssl_verification | | True | `Bool` | Validate SSL certificates (1) or ignore (0). [values: `1` (default), `0`] |
| | timeout | | 10 | `Int` | Timeout for the Proxmox API in sec. |
| | retries | | 1 | `Int` | How often a connection attempt to the defined API host should be performed. |
| | wait_time | | 1 | `Int` | How many seconds should be waited before performing another connection attempt to the API host. |
| `proxmox_cluster` | | | | | |
| | maintenance_nodes | | ['virt66.example.com'] | `List` | A list of Proxmox nodes that are defined to be in a maintenance. (must be the same node names as used within the cluster) |
| | ignore_nodes | | [] | `List` | A list of Proxmox nodes that are defined to be ignored. |
| | overprovisioning | | False | `Bool` | Avoids balancing when nodes would become overprovisioned. |
| `balancing` | | | | | |
| | enable | | True | `Bool` | Enables the guest balancing.|
| | enforce_affinity | | True | `Bool` | Enforcing affinity/anti-affinity rules but balancing might become worse. |
| | parallel | | False | `Bool` | If guests should be moved in parallel or sequentially.|
| | parallel_jobs | | 5 | `Int` | The amount if parallel jobs when migrating guests. (default: `5`)|
| | live | | True | `Bool` | If guests should be moved live or shutdown.|
| | with_local_disks | | True | `Bool` | If balancing of guests should include local disks.|
| | with_conntrack_state | | True | `Bool` | If balancing of guests should including the conntrack state.|
| | balance_types | | ['vm', 'ct'] | `List` | Defined the types of guests that should be honored. [values: `vm`, `ct`]|
| | max_job_validation | | 1800 | `Int` | How long a job validation may take in seconds. (default: 1800) |
| | balanciness | | 10 | `Int` | The maximum delta of resource usage between node with highest and lowest usage. |
| | memory_threshold | | 75 | `Int` | The maximum threshold (in percent) that needs to be hit to perform balancing actions. (Optional) |
| | method | | memory | `Str` | The balancing method that should be used. [values: `memory` (default), `cpu`, `disk`]|
| | mode | | used | `Str` | The balancing mode that should be used. [values: `used` (default), `assigned`, `psi` (pressure)] |
| | psi | | { nodes: { memory: { pressure_full: 0.20, pressure_some: 0.20, pressure_spikes: 1.00 }}} | `Dict` | A dict of PSI based thresholds for nodes and guests |
| | pools | | pools: { dev: { type: affinity }, de-nbg01-db: { type: anti-affinity }} | `Dict` | A dict of pool names and their type for creating affinity/anti-affinity rules |
| `service` | | | | | |
| | daemon | | True | `Bool` | If daemon mode should be activated. |
| | `schedule` | | | `Dict` | Schedule config block for rebalancing. |
| | | interval | 12 | `Int` | How often rebalancing should occur in daemon mode.|
| | | format | hours | `Str` | Sets the time format. [values: `hours` (default), `minutes`]|
| | `delay` | | | `Dict` | Schedule config block for an optional delay until the service starts. |
| | | enable | False | `Bool` | If a delay time should be validated.|
| | | time | 1 | `Int` | Delay time until the service starts after the initial execution.|
| | | format | hours | `Str` | Sets the time format. [values: `hours` (default), `minutes`]|
| | log_level | | INFO | `Str` | Defines the default log level that should be logged. [values: `INFO` (default), `WARNING`, `CRITICAL`, `DEBUG`] |
| Section | Option | Example | Type | Description |
|------|:------:|:------:|:------:|:------:|
| `proxmox_api` | | | | |
| | hosts | ['virt01.example.com', '10.10.10.10', 'fe01::bad:code::cafe'] | `List` | List of Proxmox nodes. Can be IPv4, IPv6 or mixed. |
| | user | root@pam | `Str` | Username for the API. |
| | pass | FooBar | `Str` | Password for the API. |
| | ssl_verification | True | `Bool` | Validate SSL certificates (1) or ignore (0). (default: 1, type: bool) |
| | timeout | 10 | `Int` | Timeout for the Proxmox API in sec. (default: 10) |
| `proxmox_cluster` | | | | |
| | maintenance_nodes | ['virt66.example.com'] | `List` | A list of Proxmox nodes that are defined to be in a maintenance. (default: []) |
| | ignore_nodes | [] | `List` | A list of Proxmox nodes that are defined to be ignored. (default: []) |
| | overprovisioning | False | `Bool` | Avoids balancing when nodes would become overprovisioned. |
| `balancing` | | | | |
| | enable | True | `Bool` | Enables the guest balancing. (default: True)|
| | force | True | `Bool` | Enforcing affinity/anti-affinity rules but balancing might become worse. (default: False) |
| | parallel | False | `Bool` | If guests should be moved in parallel or sequentially. (default: False)|
| | live | True | `Bool` | If guests should be moved live or shutdown. (default: True)|
| | with_local_disks | True | `Bool` | If balancing of guests should include local disks (default: True)|
| | balance_types | ['vm', 'ct'] | `List` | Defined the types of guests that should be honored. (default: ['vm', 'ct']) |
| | max_job_validation | 1800 | `Int` | How long a job validation may take in seconds. (default: 1800) |
| | balanciness | 10 | `Int` | The maximum delta of resource usage between node with highest and lowest usage. (default: 10) |
| | method | memory | `Str` | The balancing method that should be used. (default: memory | choices: memory, cpu, disk)|
| | mode | used | `Str` | The balancing mode that should be used. (default: used | choices: used, assigned)|
| `service` | | | | |
| | daemon | False | `Bool` | If daemon mode should be activated (default: False)|
| | schedule | 12 | `Int` | How often rebalancing should occur in hours in daemon mode (default: 12)|
| | log_level | INFO | `Str` | Defines the default log level that should be logged. (default: INFO) |
An example of the configuration file looks like:
```
proxmox_api:
hosts: ['virt01.example.com', '10.10.10.10', 'fe01:bad:code::cafe']
hosts: ['virt01.example.com', '10.10.10.10', 'fe01::bad:code::cafe']
user: root@pam
pass: crazyPassw0rd!
# API Token method
# token_id: proxlb
# token_secret: 430e308f-1337-1337-beef-1337beefcafe
ssl_verification: True
ssl_verification: False
timeout: 10
# API Connection retries
# retries: 1
# wait_time: 1
proxmox_cluster:
maintenance_nodes: ['virt66.example.com']
@@ -323,65 +180,20 @@ proxmox_cluster:
balancing:
enable: True
enforce_affinity: False
force: False
parallel: False
live: True
with_local_disks: True
with_conntrack_state: True
balance_types: ['vm', 'ct']
max_job_validation: 1800
memory_threshold: 75
balanciness: 5
method: memory
mode: used
# # PSI thresholds only apply when using mode 'psi'
# # PSI based balancing is currently in beta and req. PVE >= 9
# psi:
# nodes:
# memory:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# cpu:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# disk:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# guests:
# memory:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# cpu:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# disk:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
pools:
dev:
type: affinity
de-nbg01-db
type: anti-affinity
pin:
- virt66
- virt77
mode: assigned
service:
daemon: True
schedule:
interval: 12
format: hours
delay:
enable: False
time: 1
format: hours
log_level: INFO
daemon: False
schedule: 12
log_level: DEBUG
```
### Parameters
@@ -389,7 +201,7 @@ The following options and parameters are currently supported:
| Option | Long Option | Description | Default |
|------|:------:|------:|------:|
| -c | --config | Path to a config file. | /etc/proxlb/proxlb.yaml (default) |
| -c | --config | Path to a config file. | /etc/proxlb/proxlb.conf (default) |
| -d | --dry-run | Performs a dry-run without doing any actions. | False |
| -j | --json | Returns a JSON of the VM movement. | False |
| -b | --best-node | Returns the best next node for a VM/CT placement (useful for further usage with Terraform/Ansible). | False |
@@ -401,33 +213,19 @@ ProxLB provides an advanced mechanism to define affinity and anti-affinity rules
ProxLB implements affinity and anti-affinity rules through a tag-based system within the Proxmox web interface. Each guest (virtual machine or container) can be assigned specific tags, which then dictate its placement behavior. This method maintains a streamlined and secure approach to managing VM relationships while preserving Proxmoxs inherent permission model.
### Affinity Rules
<img align="left" src="https://cdn.gyptazy.com/img/proxlb-affinity-rules.jpg"/> Affinity rules are used to group certain VMs together, ensuring that they run on the same host whenever possible. This can be beneficial for workloads requiring low-latency communication, such as clustered databases or application servers that frequently exchange data. In general, there're two ways to manage affinity rules:
<img align="left" src="https://cdn.gyptazy.com/images/proxlb-affinity-rules.jpg"/> Affinity rules are used to group certain VMs together, ensuring that they run on the same host whenever possible. This can be beneficial for workloads requiring low-latency communication, such as clustered databases or application servers that frequently exchange data.
#### Affinity Rules by Tags
To define an affinity rule which keeps all guests assigned to this tag together on a node, users assign a tag with the prefix `plb_affinity_$TAG`:
#### Example for Screenshot
```
plb_affinity_talos
```
As a result, ProxLB will attempt to place all VMs with the `plb_affinity_web` tag on the same host (see also the attached screenshot with the same node).
#### Affinity Rules by Pools
Antoher approach is by using pools in Proxmox. This way, it can easily also combined with other resources like backup jobs. However, in this approach you need to modify the ProxLB config file to your needs. Within the `balancing` section you can create a dict of pools, including the pool name and the affinity type. Please see the example for further details:
**Example Config**
```
balancing:
[...]
pools: # Optional: Define affinity/anti-affinity rules per pool
dev: # Pool name: dev
type: affinity # Type: affinity (keeping VMs together)
pin: # Pin VMs to Nodes
- virt77 # Pinning to 'virt77' which is maybe an older system for dev labs
```
### Anti-Affinity Rules by Tags
<img align="left" src="https://cdn.gyptazy.com/img/proxlb-anti-affinity-rules.jpg"/> Conversely, anti-affinity rules ensure that designated VMs do not run on the same physical host. This is particularly useful for high-availability setups, where redundancy is crucial. Ensuring that critical services are distributed across multiple hosts reduces the risk of a single point of failure. In general, there're two ways to manage anti-affinity rules:
### Anti-Affinity Rules
<img align="left" src="https://cdn.gyptazy.com/images/proxlb-anti-affinity-rules.jpg"/> Conversely, anti-affinity rules ensure that designated VMs do not run on the same physical host. This is particularly useful for high-availability setups, where redundancy is crucial. Ensuring that critical services are distributed across multiple hosts reduces the risk of a single point of failure.
To define an anti-affinity rule that ensures to not move systems within this group to the same node, users assign a tag with the prefix:
@@ -438,85 +236,22 @@ plb_anti_affinity_ntp
As a result, ProxLB will try to place the VMs with the `plb_anti_affinity_ntp` tag on different hosts (see also the attached screenshot with the different nodes).
#### Anti-Affinity Rules by Pools
Antoher approach is by using pools in Proxmox. This way, it can easily also combined with other resources like backup jobs. However, in this approach you need to modify the ProxLB config file to your needs. Within the `balancing` section you can create a dict of pools, including the pool name and the affinity type. Please see the example for further details:
**Example Config**
```
balancing:
[...]
pools: # Optional: Define affinity/anti-affinity rules per pool
de-nbg01-db: # Pool name: de-nbg01-db
type: anti-affinity # Type: anti-affinity (spreading VMs apart)
```
**Note:** While this ensures that ProxLB tries distribute these VMs across different physical hosts within the Proxmox cluster this may not always work. If you have more guests attached to the group than nodes in the cluster, we still need to run them anywhere. If this case occurs, the next one with the most free resources will be selected.
### Ignore VMs
<img align="left" src="https://cdn.gyptazy.com/img/proxlb-ignore-vm-movement.jpg"/> Guests, such as VMs or CTs, can also be completely ignored. This means, they won't be affected by any migration (even when (anti-)affinity rules are enforced). To ensure a proper resource evaluation, these guests are still collected and evaluated but simply skipped for balancing actions. Another thing is the implementation. While ProxLB might have a very restricted configuration file including the file permissions, this file is only read- and writeable by the Proxmox administrators. However, we might have user and groups who want to define on their own that their systems shouldn't be moved. Therefore, these users can simpy set a specific tag to the guest object - just like the (anti)affinity rules.
To define a guest to be ignored from the balancing, users assign a tag with the prefix `plb_ignore_$TAG`:
#### Example for Screenshot
```
plb_ignore_dev
```
As a result, ProxLB will not migrate this guest with the `plb_ignore_dev` tag to any other node.
**Note:** Ignored guests are really ignored. Even by enforcing affinity rules this guest will be ignored.
### Pin VMs to Specific Hypervisor Nodes
<img align="left" src="https://cdn.gyptazy.com/img/proxlb-tag-node-pinning.jpg"/> Guests, such as VMs or CTs, can also be pinned to specific (and multiple) nodes in the cluster. This might be usefull when running applications with some special licensing requirements that are only fulfilled on certain nodes. It might also be interesting, when some physical hardware is attached to a node, that is not available in general within the cluster.
#### Pinning VMs to (a) specific Hypervisor Node(s) by Tag
To pin a guest to a specific cluster node, users assign a tag with the prefix `plb_pin_$nodename` to the desired guest:
#### Example for Screenshot
```
plb_pin_node03
```
As a result, ProxLB will pin the guest `dev-vm01` to the node `virt03`.
#### Pinning VMs to (a) specific Hypervisor Node(s) by Pools
Beside the tag approach, you can also pin a resource group to a specific hypervisor or groups of hypervisors by defining a `pin` key of type list.
**Example Config**
```
balancing:
[...]
pools: # Optional: Define affinity/anti-affinity rules per pool
dev: # Pool name: dev
type: affinity # Type: affinity (keeping VMs together)
pin: # Pin VMs to Nodes
- virt77 # Pinning to 'virt77' which is maybe an older system for dev labs
```
You can also repeat this step multiple times for different node names to create a potential group of allowed hosts where a the guest may be served on. In this case, ProxLB takes the node with the lowest used resources according to the defined balancing values from this group.
**Note:** The given node names from the tag are validated. This means, ProxLB validated if the given node name is really part of the cluster. In case of a wrongly defined or unavailable node name it continous to use the regular processes to make sure the guest keeps running.
## Maintenance
The `maintenance_nodes` option allows operators to designate one or more Proxmox nodes for maintenance mode. When a node is set to maintenance, no new guest workloads will be assigned to it, and all existing workloads will be migrated to other available nodes within the cluster. This process ensures that (anti)-affinity rules and resource availability are respected, preventing disruptions while maintaining optimal performance across the infrastructure.
<img src="https://cdn.gyptazy.com/images/proxlb-rebalancing-demo.gif"/>
### Adding / Removing Nodes from Maintenance
Within the section `proxmox_cluster` you can define the key `maintenance_nodes` as a list object. Simply add/remove one or more nodes with their equal name in the cluster and restart the daemon.
```
proxmox_cluster:
maintenance_nodes: ['virt66.example.com']
```
Afterwards, all guest objects will be moved to other nodes in the cluster by ensuring the best balancing.
The `maintenance_nodes` option allows operators to designate one or more Proxmox nodes for maintenance mode. When a node is set to maintenance, no new guest workloads will be assigned to it, and all existing workloads will be migrated to other available nodes within the cluster. This process ensures that (anti)-affinity rules and resource availability are respected, preventing disruptions while maintaining optimal performance across the infrastructure.
## Misc
### Bugs
Bugs can be reported via the GitHub issue tracker [here](https://github.com/gyptazy/ProxLB/issues). You may also report bugs via email or deliver PRs to fix them on your own. Therefore, you might also see the contributing chapter.
### Contributing
Feel free to add further documentation, to adjust already existing one or to contribute with code. Please take care about the style guide and naming conventions. You can find more in our [CONTRIBUTING.md](https://github.com/gyptazy/ProxLB/blob/main/CONTRIBUTING.md) file.
Feel free to add further documentation, to adjust already existing one or to contribute with code. Please take care about the style guide and naming conventions. You can find more in our [CONTRIBUTING.md](https://github.com/gyptazy/ProxLB/blob/development/CONTRIBUTING.md) file.
### Documentation
You can also find additional and more detailed documentation within the [docs/](https://github.com/gyptazy/ProxLB/tree/development/docs) directory.
### Support
If you need assistance or have any questions, we offer support through our dedicated [chat room](https://matrix.to/#/#proxlb:gyptazy.com) in Matrix or [Discord](https://discord.gg/JemGu7WbfQ). Join our community for real-time help, advice, and discussions. The Matrix and Discord room are bridged to ensure that the communication is not splitted - so simply feel free to join which fits most to you!
@@ -532,18 +267,5 @@ Connect with us in our dedicated chat room for immediate support and live intera
**Note:** Please always keep in mind that this is a one-man show project without any further help. This includes coding, testing, packaging and all the infrastructure around it to keep this project up and running.
### Enterprise-Support
Running critical infrastructure in an enterprise environment often comes with requirements that go far beyond functionality alone. Enterprises typically expect predictable service levels, defined escalation paths, and guaranteed response times. In many cases, organizations also demand 24x7 support availability to ensure that their systems remain stable and resilient, even under unexpected circumstances.
As the creator and maintainer of ProxLB, I operate as a one-man project. While I am continuously working to improve the software, I cannot provide the type of enterprise-grade support that large organizations may require. To address this need, several companies have stepped in to offer professional services around ProxLB in Proxmox VE clusters.
Below is a list of organizations currently known to provide enterprise-level support for ProxLB. If your business relies on ProxLB in production and you require more than community-based support, these providers may be a good fit for your needs:
| Company| Country | Web |
|------|:------:|:------:|
| credativ | DE | [credativ.de](https://www.credativ.de/en/portfolio/support/proxmox-virtualization/) |
*Note: If you provide support for ProxLB, feel free to create PR with your addition.*
### Author(s)
* Florian Paul Azim Hoberg @gyptazy (https://gyptazy.com)
* Florian Paul Azim Hoberg @gyptazy (https://gyptazy.com)

View File

@@ -1,15 +1,9 @@
proxmox_api:
hosts: ['virt01.example.com', '10.10.10.10', 'fe01:bad:code::cafe']
hosts: ['virt01.example.com', '10.10.10.10', 'fe01::bad:code::cafe']
user: root@pam
pass: crazyPassw0rd!
# API Token method
# token_id: proxlb
# token_secret: 430e308f-1337-1337-beef-1337beefcafe
ssl_verification: True
ssl_verification: False
timeout: 10
# API Connection retries
# retries: 1
# wait_time: 1
proxmox_cluster:
maintenance_nodes: ['virt66.example.com']
@@ -18,64 +12,17 @@ proxmox_cluster:
balancing:
enable: True
enforce_affinity: False
force: False
parallel: False
# If running parallel job, you can define
# the amount of prallel jobs (default: 5)
parallel_jobs: 1
live: True
with_local_disks: True
with_conntrack_state: True
balance_types: ['vm', 'ct'] # 'vm' | 'ct'
max_job_validation: 1800 # Maximum time (in seconds) a job validation may take
memory_threshold: 75 # Optional: Maximum threshold (in percent) to trigger balancing actions
balanciness: 5 # Maximum delta of resource usage between highest and lowest usage node
method: memory # 'memory' | 'cpu' | 'disk'
mode: used # 'assigned' | 'used' | 'psi'
# # PSI thresholds only apply when using mode 'psi'
# psi:
# nodes:
# memory:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# cpu:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# disk:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# guests:
# memory:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# cpu:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
# disk:
# pressure_full: 0.20
# pressure_some: 0.20
# pressure_spikes: 1.00
pools: # Optional: Define affinity/anti-affinity rules per pool
dev: # Pool name: dev
type: affinity # Type: affinity (keeping VMs together)
de-nbg01-db: # Pool name: de-nbg01-db
type: anti-affinity # Type: anti-affinity (spreading VMs apart)
pin: # Define a pinning og guests to specific node(s)
- virt66
- virt77
balance_types: ['vm', 'ct']
max_job_validation: 1800
balanciness: 5
method: memory
mode: assigned
service:
daemon: True
schedule:
interval: 12
format: hours
delay:
enable: False
time: 1
format: hours
log_level: INFO
daemon: False
schedule: 12
log_level: DEBUG

111
debian/changelog vendored
View File

@@ -1,111 +0,0 @@
proxlb (1.1.10) stable; urgency=medium
* Prevent redundant rebalancing by validating existing affinity enforcement before taking actions. (Closes: #335)
* Add safety-guard for PVE 8 users when activating conntrack-aware migrations mistakenly. (Closes: #359)
* Fix the Proxmox API connection validation which returned a false-positive logging message of timeouts. (Closes: #361)
* Refactored the whole Proxmox API connection function. (Closes: #361)
* Fix a crash during PVE resource pool enumeration by skipping members not having a 'name' property. (Closes: #368)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Tue, 25 Nov 2025 09:12:04 +0001
proxlb (1.1.9.1) stable; urgency=medium
* Fix quoting in f-strings which may cause issues on PVE 8 / Debian Bookworm systems. (Closes: #352)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Thu, 30 Oct 2025 17:41:02 +0001
proxlb (1.1.9) stable; urgency=medium
* Add pressure (PSI) based balancing for memory, cpu, disk (req. PVE9 or greater). (Closes: #339)
* Add (memory) threshold for nodes before running balancing. (Closes: #342)
* Add affinity/anti-affinity support by pools. (Closes: #343)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Thu, 30 Oct 2025 06:58:43 +0001
proxlb (1.1.8) stable; urgency=medium
* Fix API errors when using conntrack aware migration with older PVE version. (Closes: #318)
* Add a static ProxLB prefix to the log output when used by journal handler. (Closes: #329)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Thu, 09 Oct 2025 09:04:13 +0002
proxlb (1.1.7) stable; urgency=medium
* Add conntrack state aware migrations of VMs. (Closes: #305)
* Add graceful shutdown for SIGINT command. (Closes: #304)
* Fix crash when validating absent migration job ids. (Closes: #308)
* Fix guest object names are not being evaluated in debug log. (Closes: #310)
* Note: Have a great Dutch Proxmox Day 2025!
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Thu, 04 Sep 2025 19:23:51 +0000
proxlb (1.1.6.1) stable; urgency=medium
* Validate for node presence when pinning VMs to avoid crashing. (Closes: #296)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Thu, 04 Sep 2025 19:23:51 +0000
proxlb (1.1.6) stable; urgency=medium
* Add validation for provided API user token id to avoid confusions. (Closes: #291)
* Fix stacktrace output when validating permissions on non existing users in Proxmox. (Closes: #291)
* Fix Overprovisioning first node if anti_affinity_group has only one member. (Closes: #295)
* Validate for node presence when pinning guests to avoid crashing. (Closes: #296)
* Fix balancing evaluation of guest types (e.g., VM or CT). (Closes: #268)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Thu, 04 Sep 2025 05:12:19 +0000
proxlb (1.1.5) stable; urgency=medium
* Allow custom API ports instead of fixed tcp/8006. (Closes: #260)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Mon, 14 Jul 2025 11:07:34 +0000
proxlb (1.1.4) stable; urgency=medium
* Allow pinning of guests to a group of nodes. (Closes: #245)
* Modified log levels to make output lighter at INFO level. (Closes: #255)
* ixed an issue where balancing was performed in combination of deactivated balancing and dry-run mode. (Closes: #248)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Fri, 27 Jun 2025 16:22:58 +0000
proxlb (1.1.3) stable; urgency=medium
* Add relaod (SIGHUP) function to ProxLB to reload the configuration. (Closes: #189)
* Add optional wait time parameter to delay execution until the service takes action. (Closes: #239)
* Make the amount of parallel migrations configurable. (Closes: #241)
* Use the average CPU consumption of a guest within the last 60 minutes instead of the current CPU usage. (Closes: #94)
* Align maintenance mode with Proxmox HA maintenance mode. (Closes: #232)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Thu, 19 Jun 2025 09:10:43 +0000
proxlb (1.1.2) stable; urgency=medium
* Add a configurable retry mechanism when connecting to the Proxmox API. (Closed: #157)
* Add 1-to-1 relationships between guest and hypervisor node to ping a guest on a node. (Closes #218)
* Force type cast cpu count of guests to int for some corner cases where a str got returned. (Closed #222)
* Fix systemd unit file to run after network on non PVE nodes. (Closes #137)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Mon, 13 May 2025 18:12:04 +0000
proxlb (1.1.1) stable; urgency=medium
* Fix tag evluation for VMs for being ignored for further balancing. (Closes: #163)
* Improve logging verbosity of messages that had a wrong servity. (Closes: #165)
* Providing the API upstream error message when migration fails in debug mode (Closes: #205)
* Change the default behaviour of the daemon mode to active. (Closes: #176)
* Change the default banalcing mode to used instead of assigned. (Closes: #180)
* Set cpu_used to the cpu usage, which is a percent, times the total number of cores to get a number where guest cpu_used can be added to nodes cpu_used and be meaningful. (Closes: #195)
* Honor the value when balancing should not be performed and stop balancing. (Closes: #174)
* Allow the use of minutes instead of hours and only accept hours or minutes in the format. (Closes: #187)
* Remove hard coded memory usage from lowest usage node and use method and mode specified in configuration instead. (Closes: #197)
* Fix the guest type relationship in the logs when a migration job failed. (Closes: #204)
* Requery a guest if that running guest reports 0 cpu usage. (Closes: #200)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Sat, 20 Apr 2025 20:55:02 +0000
proxlb (1.1.0) stable; urgency=medium
* Refactored code base of ProxLB. (Closes: #114)
-- Florian Paul Azim Hoberg <gyptazy@gyptazy.com> Mon, 17 Mar 2025 18:55:02 +0000

12
debian/control vendored
View File

@@ -1,12 +0,0 @@
Source: proxlb
Maintainer: Florian Paul Azim Hoberg <gyptazy@gyptazy.com>
Section: admin
Priority: optional
Standards-Version: 4.5.0
Build-Depends: debhelper-compat (= 13), dh-python, python3-all, python3-setuptools
Package: proxlb
Architecture: all
Depends: ${python3:Depends}, ${misc:Depends}, python3-requests, python3-urllib3, python3-packaging, python3-proxmoxer, python3-yaml
Description: An advanced resource scheduler and load balancer for Proxmox clusters
An advanced resource scheduler and load balancer for Proxmox clusters that also supports maintenance mode and affinity/anti-affinity rules.

2
debian/install vendored
View File

@@ -1,2 +0,0 @@
proxlb /usr/lib/python3/dist-packages/
service/proxlb.service /lib/systemd/system/

16
debian/postinst vendored
View File

@@ -1,16 +0,0 @@
#!/bin/bash
set -e
#DEBHELPER#
if [ "$1" = "configure" ]; then
systemctl enable proxlb.service
systemctl restart proxlb.service || true
# Create the 'plb' user if it does not exist
if ! id "plb" &>/dev/null; then
useradd --system --home /var/lib/proxlb --create-home --shell /usr/sbin/nologin --group nogroup plb
echo "User 'plb' created."
else
echo "User 'plb' already exists, skipping creation."
fi
fi

16
debian/prerm vendored
View File

@@ -1,16 +0,0 @@
#!/bin/bash
set -e
#DEBHELPER#
if [ "$1" = "remove" ]; then
systemctl stop proxlb.service || true
systemctl disable proxlb.service || true
# Remove the 'plb' user if it exists
if id "plb" &>/dev/null; then
userdel --remove plb
echo "User 'plb' removed."
else
echo "User 'plb' does not exist, skipping removal."
fi
fi

4
debian/rules vendored
View File

@@ -1,4 +0,0 @@
#!/usr/bin/make -f
%:
dh $@ --with python3 --buildsystem=pybuild

View File

@@ -1 +0,0 @@
3.0 (native)

View File

@@ -1,65 +0,0 @@
# Table of Contents
- [Requirements](#requirements)
- [Where To Run?](#where-to-run)
## Requirements
ProxLB is a sophisticated load balancer designed to enhance the management and distribution of workloads within a Proxmox cluster. By fully utilizing the Proxmox API, ProxLB eliminates the need for additional SSH access, streamlining cluster management while maintaining robust security. This chapter outlines the general requirements necessary to deploy and operate ProxLB effectively.
### Proxmox Cluster Requirements
To use ProxLB, you must have an existing Proxmox cluster consisting of at least two nodes. While traditional load balancers often struggle to manage minimal node configurations, ProxLB is optimized to provide efficient load distribution even in a two-node environment. The more nodes present in the cluster, the better ProxLB can optimize resource usage and manage workloads.
### ProxLB Package Requirements
Next to the previously mentioned requirements, ProxLB also requires you to fit the following ones:
* Python3.x
* proxmoxer
* requests
* urllib3
* pyyaml
### Seamless API Integration
ProxLB relies exclusively on the Proxmox API for all management tasks. This eliminates the need for direct SSH access, ensuring a cleaner and more secure interaction with the cluster. The API integration allows ProxLB to:
- Monitor cluster health and node resource utilization
- Migrate virtual machines (VMs) and containers as needed
- Manage storage utilization and distribution
- Implement load balancing policies
### Authentication and Security Standards
ProxLB fully supports Proxmoxs integrated user management system, providing robust authentication and access control. Key features include:
- **Multi-Factor Authentication (MFA):** Enhances security by requiring multiple verification methods.
- **API Key Support:** ProxLB can utilize API keys for authentication instead of traditional username/password combinations, minimizing exposure to credentials.
- **Role-Based Access Control (RBAC):** Ensures administrators have fine-grained control over user permissions.
### Flexible Storage Support
ProxLB offers versatile storage management options, supporting both local and shared storage types. It efficiently balances storage workloads across the cluster using the following storage systems:
- **Local Storage:** Direct-attached storage on each node.
- **Shared Storage:** Includes options like iSCSI, NVMeOF, and NFS for centralized storage solutions.
- **Ceph:** Integrated support for Ceph distributed storage, providing high availability and fault tolerance.
### Network Infrastructure Requirements
For optimal performance, ProxLB requires a reliable and high-speed network connection between the nodes in the cluster. Ensure that the network infrastructure meets the following criteria:
- **Low Latency:** Essential for real-time load balancing and VM migration.
- **Sufficient Bandwidth:** Adequate to handle storage access, data replication, and migration traffic.
- **Redundant Network Paths:** Recommended for increased fault tolerance and uptime.
### System Resource Allocation
ProxLB itself requires minimal system resources to operate. However, for managing larger clusters or high workloads, ensure the node running ProxLB has adequate resources available:
- **CPU:** A modern multi-core processor.
- **Memory:** At least 2 GB of RAM.
- **Storage:** Minimal disk space for configuration files and logs.
## Where To Run?
ProxLB is lightweight and flexible where it runs on nearly any environment and only needs access to your Proxmox hosts API endpoint (commonly TCP port 8006).
Therefore, you can simply run ProxLB on:
* Bare-metal Systems
* VMs (even inside the Proxmox cluster)
* Docker/Podman Container
* LXC Container
* On a Proxmox node

View File

@@ -1,164 +0,0 @@
# Table of Contents
- [Installation](#installation)
- [Requirements / Dependencies](#requirements--dependencies)
- [Debian Package](#debian-package)
- [Quick-Start](#quick-start)
- [Details](#details)
- [Debian Packages (.deb files)](#debian-packages-deb-files)
- [RedHat Package](#redhat-package)
- [Container Images / Docker](#container-images--docker)
- [Overview of Images](#overview-of-images)
- [Source](#source)
- [Traditional System](#traditional-system)
- [Container Image](#container-image)
- [Upgrading](#upgrading)
- [Upgrading from < 1.1.0](#upgrading-from--110)
- [Upgrading from >= 1.1.0](#upgrading-from--110)
## Installation
### Requirements / Dependencies
* Python3.x
* proxmoxer
* requests
* urllib3
* pyyaml
The dependencies can simply be installed with `pip` by running the following command:
```
pip install -r requirements.txt
```
*Note: Distribution packages, such like the provided `.deb` package will automatically resolve and install all required dependencies by using already packaged version from the distribution's repository. By using the Docker (container) image or Debian packages, you do not need to take any care of the requirements listed here.*
### Debian Package
ProxLB is a powerful and flexible load balancer designed to work across various architectures, including `amd64`, `arm64`, `rv64` and many other ones that support Python. It runs independently of the underlying hardware, making it a versatile choice for different environments. This chapter covers the step-by-step process to install ProxLB on Debian-based systems, including Debian clones like Ubuntu.
#### Quick-Start
You can simply use this snippet to install the repository and to install ProxLB on your system.
```bash
echo "deb https://repo.gyptazy.com/stable /" > /etc/apt/sources.list.d/proxlb.list
wget -O /etc/apt/trusted.gpg.d/proxlb.asc https://repo.gyptazy.com/repository.gpg
apt-get update && apt-get -y install proxlb
cp /etc/proxlb/proxlb_example.yaml /etc/proxlb/proxlb.yaml
# Adjust the config to your needs
vi /etc/proxlb/proxlb.yaml
systemctl start proxlb
```
Afterwards, ProxLB is running in the background and balances your cluster by your defined balancing method (default: memory).
#### Details
ProxLB provides two different repositories:
* https://repo.gyptazy.com/stable (only stable release)
* https://repo.gyptazy.com/testing (bleeding edge - not recommended)
The repository is signed and the GPG key can be found at:
* https://repo.gyptazy.com/repository.gpg
You can also simply import it by running:
```
# KeyID: 17169F23F9F71A14AD49EDADDB51D3EB01824F4C
# UID: gyptazy Solutions Repository <contact@gyptazy.com>
# SHA256: 52c267e6f4ec799d40cdbdb29fa518533ac7942dab557fa4c217a76f90d6b0f3 repository.gpg
wget -O /etc/apt/trusted.gpg.d/proxlb.asc https://repo.gyptazy.com/repository.gpg
```
*Note: The defined repositories `repo.gyptazy.com` and `repo.proxlb.de` are the same!*
#### Debian Packages (.deb files)
If you do not want to use the repository you can also find the debian packages as a .deb file on gyptazy's CDN at:
* https://cdn.gyptazy.com/files/os/debian/proxlb/
Afterwards, you can simply install the package by running:
```bash
dpkg -i proxlb_*.deb
cp /etc/proxlb/proxlb_example.yaml /etc/proxlb/proxlb.yaml
# Adjust the config to your needs
vi /etc/proxlb/proxlb.yaml
systemctl start proxlb
```
### RedHat Package
There's currently no official support for RedHat based systems. However, there's a dummy .rpm package for such systems in the pipeline which can be found here:
* https://github.com/gyptazy/ProxLB/actions/workflows/20-pipeline-build-rpm-package.yml
### Container Images / Docker
Using the ProxLB container images is straight forward and only requires you to mount the config file.
```bash
# Pull the image
docker pull cr.gyptazy.com/proxlb/proxlb:latest
# Download the config
wget -O proxlb.yaml https://raw.githubusercontent.com/gyptazy/ProxLB/refs/heads/main/config/proxlb_example.yaml
# Adjust the config to your needs
vi proxlb.yaml
# Start the ProxLB container image with the ProxLB config
docker run -it --rm -v $(pwd)/proxlb.yaml:/etc/proxlb/proxlb.yaml proxlb
```
*Note: ProxLB container images are officially only available at cr.proxlb.de and cr.gyptazy.com.*
#### Overview of Images
| Version | Image |
|------|:------:|
| latest | cr.gyptazy.com/proxlb/proxlb:latest |
| v1.1.0 | cr.gyptazy.com/proxlb/proxlb:v1.1.0 |
| v1.0.6 | cr.gyptazy.com/proxlb/proxlb:v1.0.6 |
| v1.0.5 | cr.gyptazy.com/proxlb/proxlb:v1.0.5 |
| v1.0.4 | cr.gyptazy.com/proxlb/proxlb:v1.0.4 |
| v1.0.3 | cr.gyptazy.com/proxlb/proxlb:v1.0.3 |
| v1.0.2 | cr.gyptazy.com/proxlb/proxlb:v1.0.2 |
| v1.0.0 | cr.gyptazy.com/proxlb/proxlb:v1.0.0 |
| v0.9.9 | cr.gyptazy.com/proxlb/proxlb:v0.9.9 |
### Source
ProxLB can also easily be used from the provided sources - for traditional systems but also as a Docker/Podman container image.
#### Traditional System
Setting up and running ProxLB from the sources is simple and requires just a few commands. Ensure Python 3 and the Python dependencies are installed on your system, then run ProxLB using the following command:
```bash
git clone https://github.com/gyptazy/ProxLB.git
cd ProxLB
```
Afterwards simply adjust the config file to your needs:
```bash
vi config/proxlb.yaml
```
Start ProxLB by Python3 on the system:
```bash
python3 proxlb/main.py -c config/proxlb.yaml
```
#### Container Image
Creating a container image of ProxLB is straightforward using the provided Dockerfile. The Dockerfile simplifies the process by automating the setup and configuration required to get ProxLB running in an Alpine container. Simply follow the steps in the Dockerfile to build the image, ensuring all dependencies and configurations are correctly applied. For those looking for an even quicker setup, a ready-to-use ProxLB container image is also available, eliminating the need for manual building and allowing for immediate deployment.
```bash
git clone https://github.com/gyptazy/ProxLB.git
cd ProxLB
docker build -t proxlb .
```
Afterwards simply adjust the config file to your needs:
```bash
vi config/proxlb.yaml
```
Finally, start the created container.
```bash
docker run -it --rm -v $(pwd)/proxlb.yaml:/etc/proxlb/proxlb.yaml proxlb
```
## Upgrading
### Upgrading from < 1.1.0
Upgrading ProxLB is not supported due to a fundamental redesign introduced in version 1.1.x. With this update, ProxLB transitioned from a monolithic application to a pure Python-style project, embracing a more modular and flexible architecture. This shift aimed to improve maintainability and extensibility while keeping up with modern development practices. Additionally, ProxLB moved away from traditional ini-style configuration files and adopted YAML for configuration management. This change simplifies configuration handling, reduces the need for extensive validation, and ensures better type casting, ultimately providing a more streamlined and user-friendly experience.
### Upgrading from >= 1.1.0
Uprading within the current stable versions, starting from 1.1.0, will be possible in all supported ways.

View File

@@ -1,377 +0,0 @@
# Table of Contents
1. [Authentication / User Accounts / Permissions](#authentication--user-accounts--permissions)
1. [Authentication](#authentication)
2. [Creating a Dedicated User](#creating-a-dedicated-user)
3. [Creating an API Token for a User](#creating-an-api-token-for-a-user)
4. [Required Permissions for a User](#required-permissions-for-a-user)
2. [Configuration](#configuration)
1. [Affinity & Anti-Affinity Rules](#affinity--anti-affinity-rules)
1. [Affinity Rules](#affinity-rules)
2. [Anti-Affinity Rules](#anti-affinity-rules)
3. [Affinity / Anti-Affinity Enforcing](#affinity--anti-affinity-enforcing)
4. [Ignore VMs](#ignore-vms)
5. [Pin VMs to Hypervisor Nodes](#pin-vms-to-hypervisor-nodes)
2. [API Loadbalancing](#api-loadbalancing)
3. [Ignore Host-Nodes or Guests](#ignore-host-nodes-or-guests)
4. [IPv6 Support](#ipv6-support)
5. [Logging / Log-Level](#logging--log-level)
6. [Parallel Migrations](#parallel-migrations)
7. [Run as a Systemd-Service](#run-as-a-systemd-service)
8. [SSL Self-Signed Certificates](#ssl-self-signed-certificates)
9. [Node Maintenances](#node-maintenances)
10. [Balancing Methods](#balancing-methods)
1. [Used Resources](#used-resources)
2. [Assigned Resources](#assigned-resources)
3. [Pressure (PSI) based Resources](#pressure-psi-based-resources)
## Authentication / User Accounts / Permissions
### Authentication
ProxLB supports the traditional username and password authentication method, which is familiar to many users. This method requires users to provide their credentials (username and password) to gain access to the Proxmox system. While this method is straightforward and easy to implement, it has several security limitations. Username and password combinations can be vulnerable to brute force attacks, where an attacker systematically attempts various combinations until the correct one is found. If a user's credentials are compromised through phishing, malware, or other means, the attacker can gain unauthorized access to the system. Additionally, traditional authentication does not provide granular control over permissions and access levels, potentially exposing sensitive operations to unauthorized users.
To enhance security, ProxLB supports API token authentication. API tokens are unique identifiers that are used to authenticate API requests. They offer several advantages over traditional username and password authentication. API tokens are more secure as they are typically long, random strings that are difficult to guess. They can be revoked and regenerated as needed, reducing the risk of unauthorized access. API tokens can be associated with specific user accounts that have only the required permissions, ensuring that users only have access to the resources and operations they need. Furthermore, API tokens can be used for automated scripts and applications, facilitating seamless integration with other systems and services.
When Multi-Factor Authentication (MFA) or Two-Factor Authentication (2FA) is enabled in the Proxmox cluster, the system enforces the use of API tokens for authentication. This is because traditional username and password authentication is not considered secure enough in conjunction with MFA/2FA. To ensure the highest level of security when using API tokens, follow these best practices: Use dedicated user accounts for API tokens, each with only the necessary permissions. This limits the potential impact of a compromised token. Ensure that API tokens are long, random, and unique. Avoid using easily guessable patterns or sequences. Periodically regenerate and replace API tokens to minimize the risk of long-term exposure. Store API tokens securely, using environment variables or secure vaults. Avoid hardcoding tokens in source code or configuration files. Regularly monitor and audit the usage of API tokens to detect any suspicious activity or unauthorized access.
### Creating a Dedicated User
It is advisable to avoid using the default root@pam user for balancing tasks in ProxLB. Instead, creating a dedicated user account is recommended and can be done easily. You can create a new user through the GUI, API, or CLI. While the detailed roles required for balancing are outlined in the next chapter, you can also use the following CLI commands to create a user with the necessary roles to manage Virtual Machines (VMs) and Containers (CTs):
```
pveum role add proxlb --privs Datastore.Audit,Sys.Audit,VM.Audit,VM.Migrate
pveum user add proxlb@pve --password <password>
pveum acl modify / --roles proxlb --users proxlb@pve
```
*Note: The user management can also be done on the WebUI without invoking the CLI.*
### Creating an API Token for a User
Create an API token for user proxlb@pve with token ID proxlb and deactivated privilege separation:
```
pveum user token add proxlb@pve proxlb --privsep 0
```
Afterwards, you get the token secret returned. You can now add those entries to your ProxLB config. Make sure, that you also keep the `user` parameter, next to the new token parameters.
> [!IMPORTANT]
> The parameter `pass` then needs to be **absent**! You should also take care about the privilege and authentication mechanism behind Proxmox. You might want or even might not want to use privilege separation and this is up to your personal needs and use case.
| Proxmox API | ProxLB Config | Example |
|---|---|---|
| User | [user](https://github.com/gyptazy/ProxLB/blob/main/config/proxlb_example.yaml#L3) | proxlb@pve |
| Token ID | [token_id](https://github.com/gyptazy/ProxLB/blob/main/config/proxlb_example.yaml#L6) | proxlb |
| Token Secret | [token_secret](https://github.com/gyptazy/ProxLB/blob/main/config/proxlb_example.yaml#L7) | 430e308f-1337-1337-beef-1337beefcafe |
*Note: The API token configuration can also be done on the WebUI without invoking the CLI.*
### Required Permissions for a User
To ensure that ProxLB operates effectively and securely, it is essential to assign the appropriate permissions to the user accounts responsible for managing the load balancing tasks. The following permissions are the minimum required for a user to perform essential ProxLB operations:
* `Datastore.Audit`: Grants the ability to audit and view datastore information.
* `Sys.Audit`: Allows the user to audit and view system information.
* `VM.Audit`: Enables the user to audit and view virtual machine details.
* `VM.Migrate`: Provides the permission to migrate virtual machines.
Assigning these permissions ensures that the user can access necessary information and perform critical operations related to load balancing without granting excessive privileges. This practice helps maintain a secure and efficient ProxLB environment.
## Configuration
### Affinity & Anti-Affinity Rules
ProxLB provides an advanced mechanism to define affinity and anti-affinity rules, enabling precise control over virtual machine (VM) placement. These rules help manage resource distribution, improve high availability configurations, and optimize performance within a Proxmox Virtual Environment (PVE) cluster. By leveraging Proxmoxs integrated access management, ProxLB ensures that users can only define and manage rules for guests they have permission to access.
ProxLB implements affinity and anti-affinity rules through a tag-based system within the Proxmox web interface. Each guest (virtual machine or container) can be assigned specific tags, which then dictate its placement behavior. This method maintains a streamlined and secure approach to managing VM relationships while preserving Proxmoxs inherent permission model.
#### Affinity Rules by Tags
<img align="left" src="https://cdn.gyptazy.com/img/proxlb-affinity-rules.jpg"/> Affinity rules are used to group certain VMs together, ensuring that they run on the same host whenever possible. This can be beneficial for workloads requiring low-latency communication, such as clustered databases or application servers that frequently exchange data.
To define an affinity rule which keeps all guests assigned to this tag together on a node, users assign a tag with the prefix `plb_affinity_$TAG`:
##### Example for Screenshot
```
plb_affinity_talos
```
As a result, ProxLB will attempt to place all VMs with the `plb_affinity_web` tag on the same host (see also the attached screenshot with the same node).
#### Affinity Rules by Pools
Antoher approach is by using pools in Proxmox. This way, it can easily also combined with other resources like backup jobs. However, in this approach you need to modify the ProxLB config file to your needs. Within the `balancing` section you can create a dict of pools, including the pool name and the affinity type. Please see the example for further details:
**Example Config**
```
balancing:
[...]
pools: # Optional: Define affinity/anti-affinity rules per pool
dev: # Pool name: dev
type: affinity # Type: affinity (keeping VMs together)
```
#### Anti-Affinity Rules by Tags
<img align="left" src="https://cdn.gyptazy.com/img/proxlb-anti-affinity-rules.jpg"/> Conversely, anti-affinity rules ensure that designated VMs do not run on the same physical host. This is particularly useful for high-availability setups, where redundancy is crucial. Ensuring that critical services are distributed across multiple hosts reduces the risk of a single point of failure.
To define an anti-affinity rule that ensures to not move systems within this group to the same node, users assign a tag with the prefix:
##### Example for Screenshot
```
plb_anti_affinity_ntp
```
As a result, ProxLB will try to place the VMs with the `plb_anti_affinity_ntp` tag on different hosts (see also the attached screenshot with the different nodes).
**Note:** While this ensures that ProxLB tries distribute these VMs across different physical hosts within the Proxmox cluster this may not always work. If you have more guests attached to the group than nodes in the cluster, we still need to run them anywhere. If this case occurs, the next one with the most free resources will be selected.
#### Anti-Affinity Rules by Pools
Antoher approach is by using pools in Proxmox. This way, it can easily also combined with other resources like backup jobs. However, in this approach you need to modify the ProxLB config file to your needs. Within the `balancing` section you can create a dict of pools, including the pool name and the affinity type. Please see the example for further details:
**Example Config**
```
balancing:
[...]
pools: # Optional: Define affinity/anti-affinity rules per pool
de-nbg01-db: # Pool name: de-nbg01-db
type: anti-affinity # Type: anti-affinity (spreading VMs apart)
````
### Affinity / Anti-Affinity Enforcing
When a cluster is already balanced and does not require further adjustments, enabling the enforce_affinity parameter ensures that affinity and anti-affinity rules are still respected. This parameter prioritizes the placement of guest objects according to these rules, even if it leads to slight resource imbalances or increased migration overhead. Regularly reviewing and updating these rules, along with monitoring cluster performance, helps maintain optimal performance and reliability. By carefully managing these aspects, you can create a cluster environment that meets your specific needs and maintains a good balance of resources.
```
balancing:
enforce_affinity: True
```
*Note: This may have impacts to the cluster. Depending on the created group matrix, the result may also be an unbalanced cluster.*
### Ignore VMs / CTs
<img align="left" src="https://cdn.gyptazy.com/img/proxlb-ignore-vm-movement.jpg"/> Guests, such as VMs or CTs, can also be completely ignored. This means, they won't be affected by any migration (even when (anti-)affinity rules are enforced). To ensure a proper resource evaluation, these guests are still collected and evaluated but simply skipped for balancing actions. Another thing is the implementation. While ProxLB might have a very restricted configuration file including the file permissions, this file is only read- and writeable by the Proxmox administrators. However, we might have user and groups who want to define on their own that their systems shouldn't be moved. Therefore, these users can simpy set a specific tag to the guest object - just like the (anti)affinity rules.
To define a guest to be ignored from the balancing, users assign a tag with the prefix `plb_ignore_$TAG`:
#### Example for Screenshot
```
plb_ignore_dev
```
As a result, ProxLB will not migrate this guest with the `plb_ignore_dev` tag to any other node.
**Note:** Ignored guests are really ignored. Even by enforcing affinity rules this guest will be ignored.
### Pin VMs to Specific Hypervisor Nodes
<img align="left" src="https://cdn.gyptazy.com/images/proxlb-tag-node-pinning.jpg"/> Guests, such as VMs or CTs, can also be pinned to specific nodes in the cluster. This might be usefull when running applications with some special licensing requirements that are only fulfilled on certain nodes. It might also be interesting, when some physical hardware is attached to a node, that is not available in general within the cluster.
To pin a guest to a specific cluster node, users assign a tag with the prefix `plb_pin_$nodename` to the desired guest:
#### Example for Screenshot
```
plb_pin_node03
```
As a result, ProxLB will pin the guest `dev-vm01` to the node `virt03`.
You can also repeat this step multiple times for different node names to create a potential group of allowed hosts where a the guest may be served on. In this case, ProxLB takes the node with the lowest used resources according to the defined balancing values from this group.
**Note:** The given node names from the tag are validated. This means, ProxLB validated if the given node name is really part of the cluster. In case of a wrongly defined or unavailable node name it continous to use the regular processes to make sure the guest keeps running.
### API Loadbalancing
ProxLB supports API loadbalancing, where one or more host objects can be defined as a list. This ensures, that you can even operator ProxLB without further changes when one or more nodes are offline or in a maintenance. When defining multiple hosts, the first reachable one will be picked. You can speficy custom ports in the list. There are 4 ways of defining hosts with ports:
1. Hostname of IPv4 without port (in this case the default 8006 will be used)
2. Hostname or IPv4 with port
3. IPv6 in brackets with optional port
4. IPv6 without brackets, in this case the port is assumed after last colon
```
proxmox_api:
hosts: ['virt01.example.com', '10.10.10.10', 'fe01::bad:code::cafe', 'virt01.example.com:443', '[fc00::1]', '[fc00::1]:443', 'fc00::1:8006']
```
### Ignore Host-Nodes or Guests
In managing a Proxmox environment, it's often necessary to exclude certain host nodes and guests from various operations. For host nodes, this exclusion can be achieved by specifying them in the ignore_nodes parameter within the proxmox_api chapter, effectively preventing any automated processes from interacting with these nodes. Guests, on the other hand, can be ignored by assigning them a specific tag that starts with or is equal to plb_ignore, ensuring they are omitted from any automated tasks or monitoring. By implementing these configurations, administrators can fine-tune their Proxmox management to focus only on relevant nodes and guests, optimizing operational efficiency and resource allocation.
```
proxmox_cluster:
ignore_nodes: ['node01', 'node02']
```
### IPv6 Support
Yes, ProxLB fully supports IPv6.
### Logging / Log-Level
ProxLB supports systemd for seamless service management on Linux distributions. To enable this, create a proxLB.service file in /etc/systemd/system/ from `service/proxlb.service` within this repository.
On systems without systemd, such as FreeBSD and macOS, ProxLB runs with similar configurations but logs to stdout and stderr. The logging level and verbosity can be set in the `service` section of the configuration file:
```
service:
log_level: DEBUG
```
ProxLB only support the following log levels:
* INFO
* WARNING
* CRITICAL
* DEBUG
### Parallel Migrations
By default, parallel migrations are deactivated. This means, that a guest object gets migrated and the migration job is being watched until the VM or CT got moved to a new node. However, this may take a lot of time and many environments are fast enough to handle the IO load for multiple guest objects. However, there are always corner cases and this depends on your setup. Parallel migrations can be enabled by setting `parallel` to `True` within the `balancing` chapter:
```
balancing:
parallel: False
```
### Run as a Systemd-Service
The proxlb systemd unit orchestrates the ProxLB application. ProxLB can be used either as a one-shot solution or run periodically, depending on the configuration specified in the daemon chapter of its configuration file.
```
service:
daemon: False
schedule:
interval: 12
format: hours
```
In this configuration:
* `daemon`: False indicates that the ProxLB application is not running as a daemon and will execute as a one-shot solution.
* `schedule`: 12 defines the interval for the schedule, specifying how often rebalancing should be done if running as a daemon.
* `format`: Defines the given format of schedule where you can choose between `hours` or `minutes`.
### SSL Self-Signed Certificates
If you are using SSL self-signed certificates or non-valid certificated in general and do not want to deal with additional trust levels, you may also disable the SSL validation. This may mostly be helpful for dev- & test labs.
SSL certificate validation can be disabled in the `proxmox_api` section in the config file by setting:
```
proxmox_api:
ssl_verification: False
```
*Note: Disabling SSL certificate validation is not recommended.*
### Node Maintenances
To exclude specific nodes from receiving any new workloads during the balancing process, the `maintenance_nodes` configuration option can be used. This option allows administrators to define a list of nodes that are currently undergoing maintenance or should otherwise not be used for running virtual machines or containers.
```yaml
maintenance_nodes:
- virt66.example.com
```
which can also be written as:
```yaml
maintenance_nodes: ['virt66.example.com']
```
The maintenance_nodes key must be defined as a list, even if it only includes a single node. Each entry in the list must exactly match the node name as it is known within the Proxmox VE cluster. Do not use IP addresses, alternative DNS names, or aliases—only the actual cluster node names are valid. Once a node is marked as being in maintenance mode:
* No new workloads will be balanced or migrated onto it.
* Any existing workloads currently running on the node will be migrated away in accordance with the configured balancing strategies, assuming resources on other nodes allow.
This feature is particularly useful during planned maintenance, upgrades, or troubleshooting, ensuring that services continue to run with minimal disruption while the specified node is being worked on.
## 10. Balancing Methods
ProxLB provides multiple balancing modes that define *how* resources are evaluated and compared during cluster balancing.
Each mode reflects a different strategy for determining load and distributing guests (VMs or containers) between nodes.
Depending on your environment, provisioning strategy, and performance goals, you can choose between:
| Mode | Description | Typical Use Case |
|------|--------------|------------------|
| `used` | Uses the *actual runtime resource usage* (e.g. CPU, memory, disk). | Dynamic or lab environments with frequent workload changes and tolerance for overprovisioning. |
| `assigned` | Uses the *statically defined resource allocations* from guest configurations. | Production or SLA-driven clusters that require guaranteed resources and predictable performance. |
| `psi` | Uses Linux *Pressure Stall Information (PSI)* metrics to evaluate real system contention and pressure. | Advanced clusters that require pressure-aware decisions for proactive rebalancing. |
### 10.1 Used Resources
When **mode: `used`** is configured, ProxLB evaluates the *real usage metrics* of guest objects (VMs and CTs).
It collects the current CPU, memory, and disk usage directly from the Proxmox API to determine the *actual consumption* of each guest and node.
This mode is ideal for **dynamic environments** where workloads frequently change and **overprovisioning is acceptable**. It provides the most reactive balancing behavior, since decisions are based on live usage instead of static assignment.
Typical scenarios include:
- Production environments to distribute workloads across the nodes.
- Test or development clusters with frequent VM changes.
- Clusters where resource spikes are short-lived.
- Environments where slight resource contention is tolerable.
#### Example Configuration
```yaml
balancing:
mode: used
```
### 10.2 Assigned Resources
When **mode: `assigned`** is configured, ProxLB evaluates the *provisioned or allocated resources* of each guest (VM or CT) instead of their runtime usage.
It uses data such as **CPU cores**, **memory limits**, and **disk allocations** defined in Proxmox to calculate how much of each nodes capacity is reserved.
This mode is ideal for **production clusters** where:
- Overcommitment is *not allowed or only minimally tolerated*.
- Each nodes workload is planned based on the assigned capacities.
- Administrators want predictable resource distribution aligned with provisioning policies.
Unlike the `used` mode, `assigned` focuses purely on the *declared configuration* of guests and remains stable even if actual usage varies temporarily.
Typical scenarios include:
- Enterprise environments with SLA or QoS requirements.
- Clusters where workloads are sized deterministically.
- Situations where consistent node utilization and capacity awareness are crucial.
#### Example Configuration
```yaml
balancing:
mode: assigned
```
### 10.3 Pressure (PSI) based Resources
> [!IMPORTANT]
> PSI based balancing is still in beta! If you find any bugs, please raise an issue including metrics of all nodes and affected guests. You can provide metrics directly from PVE or Grafana (via node_exporter or pve_exporter).
When **mode: `psi`** is configured, ProxLB uses the **Linux Pressure Stall Information (PSI)** interface to measure the *real-time pressure* on system resources such as **CPU**, **memory**, and **disk I/O**.
Unlike the `used` or `assigned` modes, which rely on static or average metrics, PSI provides *direct insight into how often and how long tasks are stalled* because of insufficient resources.
This enables ProxLB to make **proactive balancing decisions** — moving workloads *before* performance degradation becomes visible to the user.
**IMPORTANT**: Predicting distributing workloads is dangerous and might not result into the expected state. Therefore, ProxLB migrates only a single instance each 60 minutes to obtain new real-metrics and to validate if further changes are required. Keep in mind, that migrations are also costly and should be avoided as much as possible.
PSI metrics are available for both **nodes** and **guest objects**, allowing fine-grained balancing decisions:
- **Node-level PSI:** Detects cluster nodes under systemic load or contention.
- **Guest-level PSI:** Identifies individual guests suffering from memory, CPU, or I/O stalls.
### PSI Metrics Explained
Each monitored resource defines three pressure thresholds:
| Key | Description |
|-----|--------------|
| `pressure_some` | Indicates partial stall conditions where some tasks are waiting for a resource. |
| `pressure_full` | Represents complete stall conditions where *all* tasks are blocked waiting for a resource. |
| `pressure_spikes` | Defines short-term burst conditions that may signal saturation spikes. |
These thresholds are expressed in **percentages** and represent how much time the kernel reports stalls over specific averaging windows (e.g. 5s, 10s, 60s).
### Example Configuration
```yaml
balancing:
mode: psi
psi:
nodes:
memory:
pressure_full: 0.20
pressure_some: 0.20
pressure_spikes: 1.00
cpu:
pressure_full: 0.20
pressure_some: 0.20
pressure_spikes: 1.00
disk:
pressure_full: 0.20
pressure_some: 0.20
pressure_spikes: 1.00
guests:
memory:
pressure_full: 0.20
pressure_some: 0.20
pressure_spikes: 1.00
cpu:
pressure_full: 0.20
pressure_some: 0.20
pressure_spikes: 1.00
disk:
pressure_full: 0.20
pressure_some: 0.20
pressure_spikes: 1.00
```

View File

@@ -1,24 +0,0 @@
## Table of Contents
1. [GUI Integration](#gui-integration)
- [How to install pve-proxmoxlb-service-ui package](https://github.com/gyptazy/ProxLB/issues/44)
2. [Proxmox HA Integration](#proxmox-ha-integration)
- [Host groups: Honour HA groups](https://github.com/gyptazy/ProxLB/issues/65)
### GUI Integration
<img align="left" src="https://cdn.gyptazy.com/images/proxlb-GUI-integration.jpg"/> ProxLB can also be accessed through the Proxmox Web UI by installing the optional `pve-proxmoxlb-service-ui` package, which depends on the proxlb package. For full Web UI integration, this package must be installed on all nodes within the cluster. Once installed, a new menu item - `Rebalancing`, appears in the cluster level under the HA section. Once installed, it offers two key functionalities:
* Rebalancing VM workloads
* Migrate VM workloads away from a defined node (e.g. maintenance preparation)
**Note:** This package is currently discontinued and will be readded at a later time. See also: [#44: How to install pve-proxmoxlb-service-ui package](https://github.com/gyptazy/ProxLB/issues/44).
### Proxmox HA Integration
Proxmox HA (High Availability) groups are designed to ensure that virtual machines (VMs) remain running within a Proxmox cluster. HA groups define specific rules for where VMs should be started or migrated in case of node failures, ensuring minimal downtime and automatic recovery.
However, when used in conjunction with ProxLB, the built-in load balancer for Proxmox, conflicts can arise. ProxLB operates with its own logic for workload distribution, taking into account affinity and anti-affinity rules. While it effectively balances guest workloads, it may re-shift and redistribute VMs in a way that does not align with HA group constraints, potentially leading to unsuitable placements.
Due to these conflicts, it is currently not recommended to use both HA groups and ProxLB simultaneously. The interaction between the two mechanisms can lead to unexpected behavior, where VMs might not adhere to HA group rules after being moved by ProxLB.
A solution to improve compatibility between HA groups and ProxLB is under evaluation, aiming to ensure that both features can work together without disrupting VM placement strategies.
See also: [#65: Host groups: Honour HA groups](https://github.com/gyptazy/ProxLB/issues/65).

View File

@@ -1,6 +0,0 @@
apiVersion: v2
name: proxlb
description: A Helm chart for self-hosted ProxLB
type: application
version: "1.1.10"
appVersion: "v1.1.10"

View File

@@ -1,13 +0,0 @@
{{- define "proxlb.fullname" -}}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{ define "proxlb.labels" }}
app.kubernetes.io/name: {{ .Release.Name }}
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/version: {{ .Chart.AppVersion }}
app.kubernetes.io/component: proxlb
{{- if .Values.labels }}
{{ toYaml .Values.labels }}
{{- end }}
{{ end }}

View File

@@ -1,11 +0,0 @@
{{- if .Values.configmap.create }}
apiVersion: v1
kind: ConfigMap
metadata:
name: proxlb-config
labels:
{{- include "proxlb.labels" . | nindent 4 }}
data:
proxlb.yaml: |
{{ toYaml .Values.configmap.config | indent 4 }}
{{ end }}

View File

@@ -1,44 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}
labels:
{{- include "proxlb.labels" . | nindent 4 }}
spec:
replicas: 1 # Number of replicas cannot be more than 1
selector:
matchLabels:
{{- include "proxlb.labels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "proxlb.labels" . | nindent 8 }}
spec:
{{- with .Values.image.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
# not interacting with the k8s cluster
automountServiceAccountToken: False
containers:
- name: proxlb
image: "{{ .Values.image.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
{{- if .Values.extraArgs.dryRun }}
- --dry-run
{{- end }}
volumeMounts:
- name: config
mountPath: /etc/proxlb/proxlb.yaml
subPath: proxlb.yaml
{{ if .Values.resources }}
resources:
{{ with .Values.resources }}
{{ toYaml . | nindent 10 }}
{{ end }}
{{ end }}
volumes:
- name: config
configMap:
name: proxlb-config

View File

@@ -1,61 +0,0 @@
image:
registry: cr.gyptazy.com
repository: proxlb/proxlb
tag: v1.1.10
pullPolicy: IfNotPresent
imagePullSecrets: [ ]
resources:
limits:
cpu: "1000m"
memory: "2Gi"
requests:
cpu: "100m"
memory: "100Mi"
labels: {}
extraArgs:
dryRun: false
configmap:
create: true
config:
proxmox_api:
hosts: []
#Can be either a user or a token
# user: ""
# pass: ""
# token_id: ""
# token_secret: ""
ssl_verification: True
timeout: 10
proxmox_cluster:
maintenance_nodes: [ ]
ignore_nodes: [ ]
overprovisioning: True
balancing:
enable: True
enforce_affinity: False
parallel: False
# If running parallel job, you can define
# the amount of prallel jobs (default: 5)
parallel_jobs: 1
live: True
with_local_disks: True
with_conntrack_state: True
balance_types: [ 'vm', 'ct' ]
max_job_validation: 1800
balanciness: 5
method: memory
mode: used
service:
daemon: True
schedule:
interval: 12
format: "hours"
delay:
enable: False
time: 1
format: "hours"
log_level: INFO

View File

@@ -1,13 +1,6 @@
#!/usr/bin/env bash
VERSION="1.1.9.1"
VERSION="1.1.0-alpha"
# ProxLB
sed -i "s/^__version__ = .*/__version__ = \"$VERSION\"/" "proxlb/utils/version.py"
sed -i "s/version=\"[0-9]*\.[0-9]*\.[0-9]*\"/version=\"$VERSION\"/" setup.py
# Helm Chart
sed -i "s/^version: .*/version: \"$VERSION\"/" helm/proxlb/Chart.yaml
sed -i "s/^appVersion: .*/appVersion: \"v$VERSION\"/" helm/proxlb/Chart.yaml
sed -i "s/^tag: .*/tag: \"v$VERSION\"/" helm/proxlb/values.yaml
echo "OK: Versions have been sucessfully set to $VERSION"
echo "OK: Versions have been sucessfully set to $VERSION"

View File

@@ -7,24 +7,16 @@ perform balancing actions based on the configuration provided. It also includes
parser for handling command-line arguments and a custom logger for systemd integration.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import logging
import signal
from utils.logger import SystemdLogger
from utils.cli_parser import CliParser
from utils.config_parser import ConfigParser
from utils.proxmox_api import ProxmoxApi
from models.nodes import Nodes
from models.features import Features
from models.guests import Guests
from models.groups import Groups
from models.calculations import Calculations
from models.balancing import Balancing
from models.pools import Pools
from utils.helper import Helper
@@ -35,10 +27,6 @@ def main():
# Initialize logging handler
logger = SystemdLogger(level=logging.INFO)
# Initialize handlers
signal.signal(signal.SIGHUP, Helper.handler_sighup)
signal.signal(signal.SIGINT, Helper.handler_sigint)
# Parses arguments passed from the CLI
cli_parser = CliParser()
cli_args = cli_parser.parse_args()
@@ -51,9 +39,6 @@ def main():
# Update log level from config and fallback to INFO if not defined
logger.set_log_level(proxlb_config.get('service', {}).get('log_level', 'INFO'))
# Validate of an optional service delay
Helper.get_service_delay(proxlb_config)
# Connect to Proxmox API & create API object
proxmox_api = ProxmoxApi(proxlb_config)
@@ -61,49 +46,29 @@ def main():
proxlb_config["proxmox_api"]["pass"] = "********"
while True:
# Validate if reload signal was sent during runtime
# and reload the ProxLB configuration and adjust log level
if Helper.proxlb_reload:
logger.info("Reloading ProxLB configuration.")
proxlb_config = config_parser.get_config()
logger.set_log_level(proxlb_config.get('service', {}).get('log_level', 'INFO'))
Helper.proxlb_reload = False
# Get all required objects from the Proxmox cluster
meta = {"meta": proxlb_config}
nodes = Nodes.get_nodes(proxmox_api, proxlb_config)
pools = Pools.get_pools(proxmox_api)
guests = Guests.get_guests(proxmox_api, pools, nodes, meta, proxlb_config)
guests = Guests.get_guests(proxmox_api, nodes)
groups = Groups.get_groups(guests, nodes)
# Merge obtained objects from the Proxmox cluster for further usage
proxlb_data = {**meta, **nodes, **guests, **pools, **groups}
proxlb_data = {**meta, **nodes, **guests, **groups}
Helper.log_node_metrics(proxlb_data)
# Validate usable features by PVE versions
Features.validate_available_features(proxlb_data)
# Update the initial node resource assignments
# by the previously created groups.
Calculations.set_node_assignments(proxlb_data)
Calculations.set_node_hot(proxlb_data)
Calculations.set_guest_hot(proxlb_data)
Calculations.get_most_free_node(proxlb_data, cli_args.best_node)
Calculations.validate_affinity_map(proxlb_data)
Calculations.relocate_guests_on_maintenance_nodes(proxlb_data)
Calculations.get_balanciness(proxlb_data)
Calculations.relocate_guests(proxlb_data)
Helper.log_node_metrics(proxlb_data, init=False)
# Perform balancing actions via Proxmox API
if proxlb_data["meta"]["balancing"].get("enable", False):
if not cli_args.dry_run:
Balancing(proxmox_api, proxlb_data)
if not cli_args.dry_run:
Balancing(proxmox_api, proxlb_data)
# Validate if the JSON output should be
# printed to stdout
Helper.print_json(proxlb_data, cli_args.json)
# Validate daemon mode
Helper.get_daemon_mode(proxlb_config)

View File

@@ -1,18 +1,12 @@
"""
The Balancing class is responsible for processing workloads on Proxmox clusters.
It processes the previously generated data (held in proxlb_data) and moves guests
and other supported types across Proxmox clusters based on the defined values by an operator.
The balancing class is responsible for processing workloads on Proxmox clusters.
The previously generated data (hold in proxlb_data) will processed and guests and
other supported types will be moved across Proxmox clusters based on the defined
values by an operator.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import proxmoxer
import time
from itertools import islice
from utils.logger import SystemdLogger
from typing import Dict, Any
@@ -25,23 +19,6 @@ class Balancing:
The previously generated data (hold in proxlb_data) will processed and guests and
other supported types will be moved across Proxmox clusters based on the defined
values by an operator.
Methods:
__init__(self, proxmox_api: any, proxlb_data: Dict[str, Any]):
Initializes the Balancing class with the provided ProxLB data and initiates the rebalancing
process for guests.
exec_rebalancing_vm(self, proxmox_api: any, proxlb_data: Dict[str, Any], guest_name: str) -> None:
Executes the rebalancing of a virtual machine (VM) to a new node within the cluster. Logs the migration
process and handles exceptions.
exec_rebalancing_ct(self, proxmox_api: any, proxlb_data: Dict[str, Any], guest_name: str) -> None:
Executes the rebalancing of a container (CT) to a new node within the cluster. Logs the migration
process and handles exceptions.
get_rebalancing_job_status(self, proxmox_api: any, proxlb_data: Dict[str, Any], guest_name: str, guest_current_node: str, job_id: int, retry_counter: int = 1) -> bool:
Monitors the status of a rebalancing job on a Proxmox node until it completes or a timeout
is reached. Returns True if the job completed successfully, False otherwise.
"""
def __init__(self, proxmox_api: any, proxlb_data: Dict[str, Any]):
@@ -49,81 +26,26 @@ class Balancing:
Initializes the Balancing class with the provided ProxLB data.
Args:
proxmox_api (object): The Proxmox API client instance used to interact with the Proxmox cluster.
proxlb_data (dict): A dictionary containing data related to the ProxLB load balancing configuration.
proxlb_data (dict): The data required for balancing VMs and CTs.
"""
def chunk_dict(data, size):
"""
Splits a dictionary into chunks of a specified size.
Args:
data (dict): The dictionary to be split into chunks.
size (int): The size of each chunk.
Yields:
dict: A chunk of the original dictionary with the specified size.
"""
logger.debug("Starting: chunk_dict.")
it = iter(data.items())
for chunk in range(0, len(data), size):
yield dict(islice(it, size))
for guest_name, guest_meta in proxlb_data["guests"].items():
# Validate if balancing should be performed in parallel or sequentially.
# If parallel balancing is enabled, set the number of parallel jobs.
parallel_jobs = proxlb_data["meta"]["balancing"].get("parallel_jobs", 5)
if not proxlb_data["meta"]["balancing"].get("parallel", False):
parallel_jobs = 1
logger.debug("Balancing: Parallel balancing is disabled. Running sequentially.")
else:
logger.debug(f"Balancing: Parallel balancing is enabled. Running with {parallel_jobs} parallel jobs.")
if guest_meta["node_current"] != guest_meta["node_target"]:
guest_id = guest_meta["id"]
guest_node_current = guest_meta["node_current"]
guest_node_target = guest_meta["node_target"]
for chunk in chunk_dict(proxlb_data["guests"], parallel_jobs):
jobs_to_wait = []
# VM Balancing
if guest_meta["type"] == "vm":
self.exec_rebalancing_vm(proxmox_api, proxlb_data, guest_name)
for guest_name, guest_meta in chunk.items():
# CT Balancing
elif guest_meta["type"] == "ct":
self.exec_rebalancing_ct(proxmox_api, proxlb_data, guest_name)
# Check if the guest's target is not the same as the current node
if guest_meta["node_current"] != guest_meta["node_target"]:
# Check if the guest is not ignored and perform the balancing
# operation based on the guest type
if not guest_meta["ignore"]:
job_id = None
# VM Balancing
if guest_meta["type"] == "vm":
if 'vm' in proxlb_data["meta"]["balancing"].get("balance_types", []):
logger.debug(f"Balancing: Balancing for guest {guest_name} of type VM started.")
job_id = self.exec_rebalancing_vm(proxmox_api, proxlb_data, guest_name)
else:
logger.debug(
f"Balancing: Balancing for guest {guest_name} will not be performed. "
"Guest is of type VM which is not included in allowed balancing types.")
# CT Balancing
elif guest_meta["type"] == "ct":
if 'ct' in proxlb_data["meta"]["balancing"].get("balance_types", []):
logger.debug(f"Balancing: Balancing for guest {guest_name} of type CT started.")
job_id = self.exec_rebalancing_ct(proxmox_api, proxlb_data, guest_name)
else:
logger.debug(
f"Balancing: Balancing for guest {guest_name} will not be performed. "
"Guest is of type CT which is not included in allowed balancing types.")
# Just in case we get a new type of guest in the future
else:
logger.critical(f"Balancing: Got unexpected guest type: {guest_meta['type']}. Cannot proceed guest: {guest_meta['name']}.")
if job_id:
jobs_to_wait.append((guest_name, guest_meta["node_current"], job_id))
else:
logger.debug(f"Balancing: Guest {guest_name} is ignored and will not be rebalanced.")
# Hopefully never reaching, but should be catched
else:
logger.debug(f"Balancing: Guest {guest_name} is already on the target node {guest_meta['node_target']} and will not be rebalanced.")
# Wait for all jobs in the current chunk to complete
for guest_name, node, job_id in jobs_to_wait:
if job_id:
self.get_rebalancing_job_status(proxmox_api, proxlb_data, guest_name, node, job_id)
logger.critical(f"Balancing: Got unexpected guest type: {guest_meta['type']}. Cannot proceed guest: {guest_meta['name']}.")
def exec_rebalancing_vm(self, proxmox_api: any, proxlb_data: Dict[str, Any], guest_name: str) -> None:
"""
@@ -144,7 +66,6 @@ class Balancing:
guest_id = proxlb_data["guests"][guest_name]["id"]
guest_node_current = proxlb_data["guests"][guest_name]["node_current"]
guest_node_target = proxlb_data["guests"][guest_name]["node_target"]
job_id = None
if proxlb_data["meta"]["balancing"].get("live", True):
online_migration = 1
@@ -157,25 +78,19 @@ class Balancing:
with_local_disks = 0
migration_options = {
'target': guest_node_target,
'target': {guest_node_target},
'online': online_migration,
'with-local-disks': with_local_disks,
'with-local-disks': with_local_disks
}
# Conntrack state aware migrations are not supported in older
# PVE versions, so we should not add it by default.
if proxlb_data["meta"]["balancing"].get("with_conntrack_state", True):
migration_options['with-conntrack-state'] = 1
try:
logger.info(f"Balancing: Starting to migrate VM guest {guest_name} from {guest_node_current} to {guest_node_target}.")
logger.debug(f"Balancing: Starting to migrate guest {guest_name} of type VM.")
job_id = proxmox_api.nodes(guest_node_current).qemu(guest_id).migrate().post(**migration_options)
job = self.get_rebalancing_job_status(proxmox_api, proxlb_data, guest_name, guest_node_current, job_id)
except proxmoxer.core.ResourceException as proxmox_api_error:
logger.critical(f"Balancing: Failed to migrate guest {guest_name} of type VM due to some Proxmox errors. Please check if resource is locked or similar.")
logger.debug(f"Balancing: Failed to migrate guest {guest_name} of type VM due to some Proxmox errors: {proxmox_api_error}")
logger.critical(f"Balancing: Failed to migrate guest {guest_name} of type CT due to some Proxmox errors. Please check if resource is locked or similar.")
logger.debug("Finished: exec_rebalancing_vm.")
return job_id
def exec_rebalancing_ct(self, proxmox_api: any, proxlb_data: Dict[str, Any], guest_name: str) -> None:
"""
@@ -196,17 +111,15 @@ class Balancing:
guest_id = proxlb_data["guests"][guest_name]["id"]
guest_node_current = proxlb_data["guests"][guest_name]["node_current"]
guest_node_target = proxlb_data["guests"][guest_name]["node_target"]
job_id = None
try:
logger.info(f"Balancing: Starting to migrate CT guest {guest_name} from {guest_node_current} to {guest_node_target}.")
logger.debug(f"Balancing: Starting to migrate guest {guest_name} of type CT.")
job_id = proxmox_api.nodes(guest_node_current).lxc(guest_id).migrate().post(target=guest_node_target, restart=1)
job = self.get_rebalancing_job_status(proxmox_api, proxlb_data, guest_name, guest_node_current, job_id)
except proxmoxer.core.ResourceException as proxmox_api_error:
logger.critical(f"Balancing: Failed to migrate guest {guest_name} of type CT due to some Proxmox errors. Please check if resource is locked or similar.")
logger.debug(f"Balancing: Failed to migrate guest {guest_name} of type CT due to some Proxmox errors: {proxmox_api_error}")
logger.debug("Finished: exec_rebalancing_ct.")
return job_id
def get_rebalancing_job_status(self, proxmox_api: any, proxlb_data: Dict[str, Any], guest_name: str, guest_current_node: str, job_id: int, retry_counter: int = 1) -> bool:
"""
@@ -224,32 +137,35 @@ class Balancing:
bool: True if the job completed successfully, False otherwise.
"""
logger.debug("Starting: get_rebalancing_job_status.")
job = proxmox_api.nodes(guest_current_node).tasks(job_id).status().get()
# Parallel migrations can take a huge time and create a higher load, if not defined by an
# operator we will use a sequential mode by default
if not proxlb_data["meta"]["balancing"].get("parallel", False):
job = proxmox_api.nodes(guest_current_node).tasks(job_id).status().get()
# Watch job id until it finalizes
if job["status"] == "running":
# Do not hammer the API while
# watching the job status
time.sleep(10)
retry_counter += 1
# Watch job id until it finalizes
if job["status"] == "running":
# Do not hammer the API while
# watching the job status
time.sleep(10)
retry_counter += 1
# Run recursion until we hit the soft-limit of maximum migration time for a guest
if retry_counter < proxlb_data["meta"]["balancing"].get("max_job_validation", 1800):
logger.debug(f"Balancing: Job ID {job_id} (guest: {guest_name}) for migration is still running... (Run: {retry_counter})")
self.get_rebalancing_job_status(proxmox_api, proxlb_data, guest_name, guest_current_node, job_id, retry_counter)
else:
logger.warning(f"Balancing: Job ID {job_id} (guest: {guest_name}) for migration took too long. Please check manually.")
logger.debug("Finished: get_rebalancing_job_status.")
return False
# Run recursion until we hit the soft-limit of maximum migration time for a guest
if retry_counter < proxlb_data["meta"]["balancing"].get("max_job_validation", 1800):
logger.debug(f"Balancing: Job ID {job_id} (guest: {guest_name}) for migration is still running... (Run: {retry_counter})")
self.get_rebalancing_job_status(proxmox_api, proxlb_data, guest_name, guest_current_node, job_id, retry_counter)
else:
logger.warning(f"Balancing: Job ID {job_id} (guest: {guest_name}) for migration took too long. Please check manually.")
logger.debug("Finished: get_rebalancing_job_status.")
return False
# Validate job output for errors when finished
if job["status"] == "stopped":
# Validate job output for errors when finished
if job["status"] == "stopped":
if job["exitstatus"] == "OK":
logger.debug(f"Balancing: Job ID {job_id} (guest: {guest_name}) was successfully.")
logger.debug("Finished: get_rebalancing_job_status.")
return True
else:
logger.critical(f"Balancing: Job ID {job_id} (guest: {guest_name}) went into an error! Please check manually.")
logger.debug("Finished: get_rebalancing_job_status.")
return False
if job["exitstatus"] == "OK":
logger.debug(f"Balancing: Job ID {job_id} (guest: {guest_name}) was sucessfully.")
logger.debug("Finished: get_rebalancing_job_status.")
return True
else:
logger.critical(f"Balancing: Job ID {job_id} (guest: {guest_name}) went into an error! Please check manually.")
logger.debug("Finished: get_rebalancing_job_status.")
return False

View File

@@ -1,15 +1,9 @@
"""
The Calculations class is responsible for handling the balancing of virtual machines (VMs)
The calculation class is responsible for handling the balancing of virtual machines (VMs)
and containers (CTs) across all available nodes in a Proxmox cluster. It provides methods
to calculate the optimal distribution of VMs and CTs based on the provided data.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import sys
from typing import Dict, Any
from utils.logger import SystemdLogger
@@ -22,37 +16,6 @@ class Calculations:
The calculation class is responsible for handling the balancing of virtual machines (VMs)
and containers (CTs) across all available nodes in a Proxmox cluster. It provides methods
to calculate the optimal distribution of VMs and CTs based on the provided data.
Methods:
__init__(proxlb_data: Dict[str, Any]):
Initializes the Calculation class with the provided ProxLB data.
set_node_assignments(proxlb_data: Dict[str, Any]) -> Dict[str, Any]:
Sets the assigned resources of the nodes based on the current assigned
guest resources by their created groups as an initial base.
get_balanciness(proxlb_data: Dict[str, Any]) -> Dict[str, Any]:
Gets the balanciness for further actions where the highest and lowest
usage or assignments of Proxmox nodes are compared.
get_most_free_node(proxlb_data: Dict[str, Any], return_node: bool = False) -> Dict[str, Any]:
Gets the name of the Proxmox node in the cluster with the most free resources based on
the user-defined method (e.g., memory) and mode (e.g., used).
relocate_guests_on_maintenance_nodes(proxlb_data: Dict[str, Any]):
Relocates guests that are currently on nodes marked for maintenance to
nodes with the most available resources.
relocate_guests(proxlb_data: Dict[str, Any]):
Relocates guests within the provided data structure to ensure affinity groups are
placed on nodes with the most free resources.
val_anti_affinity(proxlb_data: Dict[str, Any], guest_name: str):
Validates and assigns nodes to guests based on anti-affinity rules.
update_node_resources(proxlb_data):
Updates the resource allocation and usage statistics for nodes when a guest
is moved from one node to another.
"""
def __init__(self, proxlb_data: Dict[str, Any]):
@@ -66,7 +29,7 @@ class Calculations:
@staticmethod
def set_node_assignments(proxlb_data: Dict[str, Any]) -> Dict[str, Any]:
"""
Set the assigned resources of the nodes based on the current assigned
Set the assigned ressources of the nodes based on the current assigned
guest resources by their created groups as an initial base.
Args:
@@ -80,9 +43,8 @@ class Calculations:
for guest_name in group_meta["guests"]:
guest_node_current = proxlb_data["guests"][guest_name]["node_current"]
# Update resource assignments
# Update Hardware assignments
# Update assigned values for the current node
logger.debug(f"set_node_assignment of guest {guest_name} on node {guest_node_current} with cpu_total: {proxlb_data['guests'][guest_name]['cpu_total']}, memory_total: {proxlb_data['guests'][guest_name]['memory_total']}, disk_total: {proxlb_data['guests'][guest_name]['disk_total']}.")
proxlb_data["nodes"][guest_node_current]["cpu_assigned"] += proxlb_data["guests"][guest_name]["cpu_total"]
proxlb_data["nodes"][guest_node_current]["memory_assigned"] += proxlb_data["guests"][guest_name]["memory_total"]
proxlb_data["nodes"][guest_node_current]["disk_assigned"] += proxlb_data["guests"][guest_name]["disk_total"]
@@ -93,83 +55,6 @@ class Calculations:
logger.debug("Finished: set_node_assignments.")
def set_node_hot(proxlb_data: Dict[str, Any]) -> Dict[str, Any]:
"""
Evaluates node 'full' pressure metrics for memory, cpu, and io
against defined thresholds and sets <metric>_pressure_hot = True
when a node is considered HOT.
Returns the modified proxlb_data dict.
"""
logger.debug("Starting: set_node_hot.")
balancing_cfg = proxlb_data.get("meta", {}).get("balancing", {})
thresholds = balancing_cfg.get("psi_thresholds", balancing_cfg.get("psi", {}).get("nodes", {}))
nodes = proxlb_data.get("nodes", {})
for node_name, node in nodes.items():
if node.get("maintenance"):
continue
if node.get("ignore"):
continue
# PSI metrics are only availavble on Proxmox VE 9.0 and higher.
if proxlb_data["meta"]["balancing"].get("mode", "used") == "psi":
if tuple(map(int, proxlb_data["nodes"][node["name"]]["pve_version"].split('.'))) < tuple(map(int, "9.0".split('.'))):
logger.critical(f"Proxmox node {node['name']} runs Proxmox VE version {proxlb_data['nodes'][node['name']]['pve_version']}."
" PSI metrics require Proxmox VE 9.0 or higher. Balancing deactivated!")
for metric, threshold in thresholds.items():
pressure_full = node.get(f"{metric}_pressure_full_percent", 0.0)
pressure_some = node.get(f"{metric}_pressure_some_percent", 0.0)
pressure_spikes = node.get(f"{metric}_pressure_full_spikes_percent", 0.0)
is_hot = (pressure_full >= threshold["pressure_full"] and pressure_some >= threshold["pressure_some"]) or (pressure_spikes >= threshold["pressure_spikes"])
if is_hot:
logger.debug(f"Set node {node['name']} as hot based on {metric} pressure metrics.")
proxlb_data["nodes"][node["name"]][f"{metric}_pressure_hot"] = True
proxlb_data["nodes"][node["name"]][f"pressure_hot"] = True
else:
logger.debug(f"Node {node['name']} is not hot based on {metric} pressure metrics.")
logger.debug("Finished: set_node_hot.")
return proxlb_data
def set_guest_hot(proxlb_data: Dict[str, Any]) -> Dict[str, Any]:
"""
Evaluates guest 'full' pressure metrics for memory, cpu, and io
against defined thresholds and sets <metric>_pressure_hot = True
when a guest is considered HOT.
Returns the modified proxlb_data dict.
"""
logger.debug("Starting: set_guest_hot.")
balancing_cfg = proxlb_data.get("meta", {}).get("balancing", {})
thresholds = balancing_cfg.get("psi_thresholds", balancing_cfg.get("psi", {}).get("guests", {}))
guests = proxlb_data.get("guests", {})
for guest_name, guest in guests.items():
if guest.get("ignore"):
continue
for metric, threshold in thresholds.items():
pressure_full = guest.get(f"{metric}_pressure_full_percent", 0.0)
pressure_some = guest.get(f"{metric}_pressure_some_percent", 0.0)
pressure_spikes = guest.get(f"{metric}_pressure_full_spikes_percent", 0.0)
is_hot = (pressure_full >= threshold["pressure_full"] and pressure_some >= threshold["pressure_some"]) or (pressure_spikes >= threshold["pressure_spikes"])
if is_hot:
logger.debug(f"Set guest {guest['name']} as hot based on {metric} pressure metrics.")
proxlb_data["guests"][guest["name"]][f"{metric}_pressure_hot"] = True
proxlb_data["guests"][guest["name"]][f"pressure_hot"] = True
else:
logger.debug(f"guest {guest['name']} is not hot based on {metric} pressure metrics.")
logger.debug("Finished: set_guest_hot.")
return proxlb_data
@staticmethod
def get_balanciness(proxlb_data: Dict[str, Any]) -> Dict[str, Any]:
"""
@@ -190,74 +75,17 @@ class Calculations:
method = proxlb_data["meta"]["balancing"].get("method", "memory")
mode = proxlb_data["meta"]["balancing"].get("mode", "used")
balanciness = proxlb_data["meta"]["balancing"].get("balanciness", 10)
if mode == "assigned":
method_value = [node_meta[f"{method}_{mode}_percent"] for node_meta in proxlb_data["nodes"].values()]
if proxlb_data["meta"]["balancing"].get(f"{method}_threshold", None):
threshold = proxlb_data["meta"]["balancing"].get(f"{method}_threshold")
highest_usage_node = max(proxlb_data["nodes"].values(), key=lambda x: x[f"{method}_{mode}_percent"])
highest_node_value = highest_usage_node[f"{method}_{mode}_percent"]
if highest_node_value >= threshold:
logger.debug(f"Guest balancing is required. Highest {method} usage node {highest_usage_node['name']} is above the defined threshold of {threshold}% with a value of {highest_node_value}%.")
proxlb_data["meta"]["balancing"]["balance"] = True
else:
logger.debug(f"Guest balancing is ok. Highest {method} usage node {highest_usage_node['name']} is below the defined threshold of {threshold}% with a value of {highest_node_value}%.")
proxlb_data["meta"]["balancing"]["balance"] = False
else:
logger.debug(f"No {method} threshold defined for balancing. Skipping threshold check.")
elif mode == "used":
method_value = [node_meta[f"{method}_{mode}_percent"] for node_meta in proxlb_data["nodes"].values()]
if proxlb_data["meta"]["balancing"].get(f"{method}_threshold", None):
threshold = proxlb_data["meta"]["balancing"].get(f"{method}_threshold")
highest_usage_node = max(proxlb_data["nodes"].values(), key=lambda x: x[f"{method}_{mode}_percent"])
highest_node_value = highest_usage_node[f"{method}_{mode}_percent"]
if highest_node_value >= threshold:
logger.debug(f"Guest balancing is required. Highest {method} usage node {highest_usage_node['name']} is above the defined threshold of {threshold}% with a value of {highest_node_value}%.")
proxlb_data["meta"]["balancing"]["balance"] = True
else:
logger.debug(f"Guest balancing is ok. Highest {method} usage node {highest_usage_node['name']} is below the defined threshold of {threshold}% with a value of {highest_node_value}%.")
proxlb_data["meta"]["balancing"]["balance"] = False
else:
logger.debug(f"No {method} threshold defined for balancing. Skipping threshold check.")
elif mode == "psi":
method_value = [node_meta[f"{method}_pressure_full_spikes_percent"] for node_meta in proxlb_data["nodes"].values()]
any_node_hot = any(node.get(f"{method}_pressure_hot", False) for node in proxlb_data["nodes"].values())
any_guest_hot = any(node.get(f"{method}_pressure_hot", False) for node in proxlb_data["guests"].values())
if any_node_hot:
logger.debug(f"Guest balancing is required. A node is marked as HOT based on {method} pressure metrics.")
proxlb_data["meta"]["balancing"]["balance"] = True
else:
logger.debug(f"Guest balancing is ok. No node is marked as HOT based on {method} pressure metrics.")
if any_guest_hot:
logger.debug(f"Guest balancing is required. A guest is marked as HOT based on {method} pressure metrics.")
proxlb_data["meta"]["balancing"]["balance"] = True
else:
logger.debug(f"Guest balancing is ok. No guest is marked as HOT based on {method} pressure metrics.")
return proxlb_data
else:
logger.critical(f"Unknown balancing mode: {mode} provided. Cannot get balanciness.")
sys.exit(1)
method_value = [node_meta[f"{method}_{mode}_percent"] for node_meta in proxlb_data["nodes"].values()]
method_value_highest = max(method_value)
method_value_lowest = min(method_value)
if method_value_highest - method_value_lowest > balanciness:
proxlb_data["meta"]["balancing"]["balance"] = True
logger.debug(f"Guest balancing is required. Highest value: {method_value_highest}, lowest value: {method_value_lowest} balanced by {method} and {mode}.")
logger.critical(f"Guest balancing is required. Highest value: {method_value_highest}, lowest value: {method_value_lowest} balanced by {method} and {mode}.")
else:
logger.debug(f"Guest balancing is ok. Highest value: {method_value_highest}, lowest value: {method_value_lowest} balanced by {method} and {mode}.")
logger.critical(f"Guest balancing is ok. Highest value: {method_value_highest}, lowest value: {method_value_lowest} balanced by {method} and {mode}.")
else:
logger.warning("No guests for balancing found.")
@@ -265,7 +93,7 @@ class Calculations:
logger.debug("Finished: get_balanciness.")
@staticmethod
def get_most_free_node(proxlb_data: Dict[str, Any], return_node: bool = False, guest_node_relation_list: list = []) -> Dict[str, Any]:
def get_most_free_node(proxlb_data: Dict[str, Any], return_node: bool = False) -> Dict[str, Any]:
"""
Get the name of the Proxmox node in the cluster with the most free resources based on
the user defined method (e.g.: memory) and mode (e.g.: used).
@@ -274,8 +102,6 @@ class Calculations:
proxlb_data (Dict[str, Any]): The data holding all content of all objects.
return_node (bool): The indicator to simply return the best node for further
assignments.
guest_node_relation_list (list): A list of nodes that have a tag on the given
guest relationship for pinning.
Returns:
Dict[str, Any]: Updated meta data section of the node with the most free resources that should
@@ -284,34 +110,9 @@ class Calculations:
logger.debug("Starting: get_most_free_node.")
proxlb_data["meta"]["balancing"]["balance_next_node"] = ""
# Filter and exclude nodes that are in maintenance mode
# Do not include nodes that are marked in 'maintenance'
filtered_nodes = [node for node in proxlb_data["nodes"].values() if not node["maintenance"]]
# Filter and include nodes that given by a relationship between guest and node. This is only
# used if the guest has a relationship to a node defined by "pin" tags.
if len(guest_node_relation_list) > 0:
filtered_nodes = [node for node in proxlb_data["nodes"].values() if node["name"] in guest_node_relation_list]
# Filter by the defined methods and modes for balancing
method = proxlb_data["meta"]["balancing"].get("method", "memory")
mode = proxlb_data["meta"]["balancing"].get("mode", "used")
if mode == "assigned":
logger.debug(f"Get best node for balancing by assigned {method} resources.")
lowest_usage_node = min(filtered_nodes, key=lambda x: x[f"{method}_{mode}_percent"])
elif mode == "used":
logger.debug(f"Get best node for balancing by used {method} resources.")
lowest_usage_node = min(filtered_nodes, key=lambda x: x[f"{method}_{mode}_percent"])
elif mode == "psi":
logger.debug(f"Get best node for balancing by pressure of {method} resources.")
lowest_usage_node = min(filtered_nodes, key=lambda x: x[f"{method}_pressure_full_spikes_percent"])
else:
logger.critical(f"Unknown balancing mode: {mode} provided. Cannot get best node.")
sys.exit(1)
lowest_usage_node = min(filtered_nodes, key=lambda x: x["memory_used_percent"])
proxlb_data["meta"]["balancing"]["balance_reason"] = 'resources'
proxlb_data["meta"]["balancing"]["balance_next_node"] = lowest_usage_node["name"]
@@ -340,7 +141,7 @@ class Calculations:
Returns:
None
"""
logger.debug("Starting: relocate_guests_on_maintenance_nodes.")
logger.debug("Starting: get_most_free_node.")
proxlb_data["meta"]["balancing"]["balance_next_guest"] = ""
for guest_name in proxlb_data["groups"]["maintenance"]:
@@ -351,7 +152,7 @@ class Calculations:
Calculations.update_node_resources(proxlb_data)
logger.warning(f"Warning: Balancing may not be perfect because guest {guest_name} was located on a node which is in maintenance mode.")
logger.debug("Finished: relocate_guests_on_maintenance_nodes.")
logger.debug("Finished: get_most_free_node.")
@staticmethod
def relocate_guests(proxlb_data: Dict[str, Any]):
@@ -369,13 +170,13 @@ class Calculations:
None
"""
logger.debug("Starting: relocate_guests.")
if proxlb_data["meta"]["balancing"]["balance"] or proxlb_data["meta"]["balancing"].get("enforce_affinity", False):
if proxlb_data["meta"]["balancing"]["balance"] or proxlb_data["meta"]["balancing"]["force"]:
if proxlb_data["meta"]["balancing"].get("balance", False):
logger.debug("Balancing of guests will be performed. Reason: balanciness")
logger.debug("Balancing of guests will be performt. Reason: balanciness")
if proxlb_data["meta"]["balancing"].get("enforce_affinity", False):
logger.debug("Balancing of guests will be performed. Reason: enforce affinity balancing")
if proxlb_data["meta"]["balancing"].get("force", False):
logger.debug("Balancing of guests will be performt. Reason: force balancing")
for group_name in proxlb_data["groups"]["affinity"]:
@@ -385,28 +186,8 @@ class Calculations:
Calculations.get_most_free_node(proxlb_data)
for guest_name in proxlb_data["groups"]["affinity"][group_name]["guests"]:
mode = proxlb_data["meta"]["balancing"].get("mode", "used")
if mode == 'psi':
logger.debug(f"Evaluating guest relocation based on {mode} mode.")
method = proxlb_data["meta"]["balancing"].get("method", "memory")
processed_guests_psi = proxlb_data["meta"]["balancing"].setdefault("processed_guests_psi", [])
unprocessed_guests_psi = [guest for guest in proxlb_data["guests"].values() if guest["name"] not in processed_guests_psi]
# Filter by the defined methods and modes for balancing
highest_usage_guest = max(unprocessed_guests_psi, key=lambda x: x[f"{method}_pressure_full_spikes_percent"])
# Append guest to the psi based processed list of guests
if highest_usage_guest["name"] == guest_name and guest_name not in proxlb_data["meta"]["balancing"]["processed_guests_psi"]:
proxlb_data["meta"]["balancing"]["processed_guests_psi"].append(guest_name)
proxlb_data["meta"]["balancing"]["balance_next_guest"] = guest_name
else:
logger.debug(f"Evaluating guest relocation based on {mode} mode.")
proxlb_data["meta"]["balancing"]["balance_next_guest"] = guest_name
proxlb_data["meta"]["balancing"]["balance_next_guest"] = guest_name
Calculations.val_anti_affinity(proxlb_data, guest_name)
Calculations.val_node_relationships(proxlb_data, guest_name)
Calculations.update_node_resources(proxlb_data)
logger.debug("Finished: relocate_guests.")
@@ -430,35 +211,30 @@ class Calculations:
None
"""
logger.debug("Starting: val_anti_affinity.")
# Start by iterating over all defined anti-affinity groups
# Start by interating over all defined anti-affinity groups
for group_name in proxlb_data["groups"]["anti_affinity"].keys():
# Validate if the provided guest is included in the anti-affinity group
# Validate if the provided guest ist included in the anti-affinity group
if guest_name in proxlb_data["groups"]["anti_affinity"][group_name]['guests'] and not proxlb_data["guests"][guest_name]["processed"]:
logger.debug(f"Anti-Affinity: Guest: {guest_name} is included in anti-affinity group: {group_name}.")
# Check if the group has only one member. If so skip new guest node assignment.
if proxlb_data["groups"]["anti_affinity"][group_name]["counter"] > 1:
logger.debug(f"Anti-Affinity: Group has more than 1 member.")
# Iterate over all available nodes
for node_name in proxlb_data["nodes"].keys():
# Iterate over all available nodes
for node_name in proxlb_data["nodes"].keys():
# Only select node if it was not used before and is not in a
# maintenance mode. Afterwards, add it to the list of already
# used nodes for the current anti-affinity group
if node_name not in proxlb_data["groups"]["anti_affinity"][group_name]["used_nodes"]:
# Only select node if it was not used before and is not in a
# maintenance mode. Afterwards, add it to the list of already
# used nodes for the current anti-affinity group
if node_name not in proxlb_data["groups"]["anti_affinity"][group_name]["used_nodes"]:
if not proxlb_data["nodes"][node_name]["maintenance"]:
# If the node has not been used yet, we assign this node to the guest
proxlb_data["meta"]["balancing"]["balance_next_node"] = node_name
proxlb_data["groups"]["anti_affinity"][group_name]["used_nodes"].append(node_name)
logger.debug(f"Node: {node_name} marked as used for anti-affinity group: {group_name} with guest {guest_name}")
break
if not proxlb_data["nodes"][node_name]["maintenance"]:
# If the node has not been used yet, we assign this node to the guest
proxlb_data["meta"]["balancing"]["balance_next_node"] = node_name
proxlb_data["groups"]["anti_affinity"][group_name]["used_nodes"].append(node_name)
logger.debug(f"Node: {node_name} marked as used for anti-affinity group: {group_name} with guest {guest_name}")
break
else:
logger.critical(f"Node: {node_name} already got used for anti-affinity group:: {group_name}. (Tried for guest: {guest_name})")
else:
logger.debug(f"Anti-Affinity: Group has less than 2 members. Skipping node calculation for the group.")
else:
logger.critical(f"Node: {node_name} already got used for anti-affinity group:: {group_name}. (Tried for guest: {guest_name})")
else:
logger.debug(f"Guest: {guest_name} is not included in anti-affinity group: {group_name}. Skipping.")
@@ -466,40 +242,7 @@ class Calculations:
logger.debug("Finished: val_anti_affinity.")
@staticmethod
def val_node_relationships(proxlb_data: Dict[str, Any], guest_name: str):
"""
Validates and assigns guests to nodes based on defined relationships based on tags.
Parameters:
proxlb_data (Dict[str, Any]): The data holding all content of all objects.
guest_name (str): The name of the guest to be validated and assigned a node.
Returns:
None
"""
logger.debug("Starting: val_node_relationships.")
proxlb_data["guests"][guest_name]["processed"] = True
if len(proxlb_data["guests"][guest_name]["node_relationships"]) > 0:
logger.debug(f"Guest '{guest_name}' has relationships defined to node(s): {','.join(proxlb_data['guests'][guest_name]['node_relationships'])}. Pinning to node.")
# Get the node with the most free resources of the group
guest_node_relation_list = proxlb_data["guests"][guest_name]["node_relationships"]
Calculations.get_most_free_node(proxlb_data, False, guest_node_relation_list)
# Validate if the specified node name is really part of the cluster
if proxlb_data["meta"]["balancing"]["balance_next_node"] in proxlb_data["nodes"].keys():
logger.debug(f"Guest '{guest_name}' has a specific relationship defined to node: {proxlb_data['meta']['balancing']['balance_next_node']} is a known hypervisor node in the cluster.")
else:
logger.warning(f"Guest '{guest_name}' has a specific relationship defined to node: {proxlb_data['meta']['balancing']['balance_next_node']} but this node name is not known in the cluster!")
else:
logger.debug(f"Guest '{guest_name}' does not have any specific node relationships.")
logger.debug("Finished: val_node_relationships.")
@staticmethod
def update_node_resources(proxlb_data: Dict[str, Any]):
def update_node_resources(proxlb_data):
"""
Updates the resource allocation and usage statistics for nodes when a guest
is moved from one node to another.
@@ -519,11 +262,6 @@ class Calculations:
"""
logger.debug("Starting: update_node_resources.")
guest_name = proxlb_data["meta"]["balancing"]["balance_next_guest"]
if guest_name == "":
logger.debug("No guest defined to update node resources for.")
return
node_current = proxlb_data["guests"][guest_name]["node_current"]
node_target = proxlb_data["meta"]["balancing"]["balance_next_node"]
@@ -568,142 +306,3 @@ class Calculations:
logger.debug(f"Set guest {guest_name} from node {node_current} to node {node_target}.")
logger.debug("Finished: update_node_resources.")
def validate_affinity_map(proxlb_data: Dict[str, Any]):
"""
Validates the affinity and anti-affinity constraints for all guests in the ProxLB data structure.
This function iterates through each guest and checks both affinity and anti-affinity rules.
If any guest violates these constraints, it sets the enforce_affinity flag to trigger rebalancing
and skips further validation for efficiency.
Args:
proxlb_data (Dict[str, Any]): A dictionary containing ProxLB configuration with the following structure:
- "guests" (list): List of guest identifiers to validate
- "meta" (dict): Metadata dictionary containing:
- "balancing" (dict): Balancing configuration with "enforce_affinity" flag
Returns:
None: Modifies proxlb_data in-place by updating the "enforce_affinity" flag in meta.balancing
Raises:
None: Function handles validation gracefully and logs outcomes
"""
logger.debug("Starting: validate_current_affinity.")
balancing_ok = True
for guest in proxlb_data["guests"]:
# We do not need to validate anymore if rebalancing is required
if balancing_ok is False:
proxlb_data["meta"]["balancing"]["enforce_affinity"] = True
logger.debug(f"Rebalancing based on affinity/anti-affinity map is required. Skipping further validation...")
break
balancing_state_affinity = Calculations.validate_current_affinity(proxlb_data, guest)
balancing_state_anti_affinity = Calculations.validate_current_anti_affinity(proxlb_data, guest)
logger.debug(f"Affinity for guest {guest} is {'valid' if balancing_state_affinity else 'NOT valid'}")
logger.debug(f"Anti-affinity for guest {guest} is {'valid' if balancing_state_anti_affinity else 'NOT valid'}")
balancing_ok = not balancing_state_affinity or not balancing_state_anti_affinity
if balancing_ok:
logger.debug(f"Rebalancing based on affinity/anti-affinity map is not required.")
proxlb_data["meta"]["balancing"]["enforce_affinity"] = False
logger.debug("Finished: validate_current_affinity.")
@staticmethod
def get_guest_node(proxlb_data: Dict[str, Any], guest_name: str) -> str:
"""
Return a currently assoicated PVE node where the guest is running on.
Args:
proxlb_data (Dict[str, Any]): A dictionary containing ProxLB configuration.
Returns:
node_name_current (str): The name of the current node where the guest runs on.
"""
return proxlb_data["guests"][guest_name]["node_current"]
@staticmethod
def validate_current_affinity(proxlb_data: Dict[str, Any], guest_name: str) -> bool:
"""
Validate that all guests in affinity groups containing the specified guest are on the same non-maintenance node.
This function checks affinity group constraints for a given guest. It ensures that:
1. All guests within an affinity group are located on the same physical node
2. The node hosting the affinity group is not in maintenance mode
Args:
proxlb_data (Dict[str, Any]): A dictionary containing the complete ProxLB state including:
- "groups": Dictionary with "affinity" key containing affinity group definitions
- "guests": Dictionary with guest information
- "nodes": Dictionary with node information including maintenance status
guest_name (str): The name of the guest to validate affinity for
Returns:
bool: True if all affinity groups containing the guest are valid (all members on same
non-maintenance node), False otherwise
"""
logger.debug("Starting: validate_current_affinity.")
for group_name, grp in proxlb_data["groups"]["affinity"].items():
if guest_name not in grp["guests"]:
continue
nodes = []
for group in grp["guests"]:
if group not in proxlb_data["guests"]:
continue
node = Calculations.get_guest_node(proxlb_data, group)
if proxlb_data["nodes"][node]["maintenance"]:
logger.debug(f"Group '{group_name}' invalid: node '{node}' in maintenance.")
return False
nodes.append(node)
if len(set(nodes)) != 1:
logger.debug(f"Group '{group_name}' invalid: guests spread across nodes {set(nodes)}.")
return False
return True
@staticmethod
def validate_current_anti_affinity(proxlb_data: Dict[str, Any], guest_name: str) -> bool:
"""
Validate that all guests in anti-affinity groups containing the specified guest are not on the same node.
This function checks anti-affinity group constraints for a given guest. It ensures that:
1. All guests within an anti-affinity group are located on the same physical node
2. The node hosting the anti-affinity group is not in maintenance mode
Args:
proxlb_data (Dict[str, Any]): A dictionary containing the complete ProxLB state including:
- "groups": Dictionary with "affinity" key containing affinity group definitions
- "guests": Dictionary with guest information
- "nodes": Dictionary with node information including maintenance status
guest_name (str): The name of the guest to validate affinity for
Returns:
bool: True if all anti-affinity groups containing the guest are valid (all members on different
non-maintenance node), False otherwise
"""
logger.debug("Starting: validate_current_anti_affinity.")
for group_name, grp in proxlb_data["groups"]["anti_affinity"].items():
if guest_name not in grp["guests"]:
continue
nodes = []
for group in grp["guests"]:
if group not in proxlb_data["guests"]:
continue
node = Calculations.get_guest_node(proxlb_data, group)
if proxlb_data["nodes"][node]["maintenance"]:
return False
nodes.append(node)
if len(nodes) != len(set(nodes)):
return False
return True

View File

@@ -1,90 +0,0 @@
"""
ProxLB Features module for validating and adjusting feature flags
based on Proxmox VE node versions and cluster compatibility.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
from typing import List
from typing import Dict, Any
from utils.logger import SystemdLogger
from packaging import version
logger = SystemdLogger()
class Features:
"""
ProxLB Features module for validating and adjusting feature flags
based on Proxmox VE node versions and cluster compatibility.
Responsibilities:
- Validate and adjust feature flags based on Proxmox VE node versions.
Methods:
__init__():
No-op initializer.
validate_available_features(proxlb_data: dict) -> None:
Static method that inspects proxlb_data["nodes"] versions and disables
incompatible balancing features for Proxmox VE versions < 9.0.0.
This function mutates proxlb_data in place.
Notes:
- Expects proxlb_data to be a dict with "nodes" and "meta" keys.
"""
def __init__(self):
"""
Initializes the Features class.
"""
@staticmethod
def validate_available_features(proxlb_data: any) -> None:
"""
Validate and adjust feature flags in the provided proxlb_data according to Proxmox VE versions.
This function inspects the cluster node versions in proxlb_data and disables features
that are incompatible with Proxmox VE versions older than 9.0.0. Concretely, if any node
reports a 'pve_version' lower than "9.0.0":
- If meta.balancing.with_conntrack_state is truthy, it is set to False and a warning is logged.
- If meta.balancing.mode equals "psi", meta.balancing.enable is set to False and a warning is logged.
proxlb_data (dict): Cluster data structure that must contain:
- "nodes": a mapping (e.g., dict) whose values are mappings containing a 'pve_version' string.
- "meta": a mapping that may contain a "balancing" mapping with keys:
- "with_conntrack_state" (bool, optional)
- "mode" (str, optional)
- "enable" (bool, optional)
None: The function mutates proxlb_data in place to disable incompatible features.
Side effects:
- Mutates proxlb_data["meta"]["balancing"] when incompatible features are detected.
- Emits debug and warning log messages.
Notes:
- Unexpected or missing keys/types in proxlb_data may raise KeyError or TypeError.
- Version comparison uses semantic version parsing; callers should provide versions as strings.
Returns:
None
"""
logger.debug("Starting: validate_available_features.")
any_non_pve9_node = any(version.parse(n['pve_version']) < version.parse("9.0.0") for n in proxlb_data["nodes"].values())
if any_non_pve9_node:
with_conntrack_state = proxlb_data["meta"].get("balancing", {}).get("with_conntrack_state", False)
if with_conntrack_state:
logger.warning("Non Proxmox VE 9 systems detected: Deactivating migration option 'with-conntrack-state'!")
proxlb_data["meta"]["balancing"]["with_conntrack_state"] = False
psi_balancing = proxlb_data["meta"].get("balancing", {}).get("mode", None)
if psi_balancing == "psi":
logger.warning("Non Proxmox VE 9 systems detected: Deactivating balancing!")
proxlb_data["meta"]["balancing"]["enable"] = False
logger.debug("Finished: validate_available_features.")

View File

@@ -1,15 +1,9 @@
"""
The Groups class is responsible for handling the correlations between the guests
and their groups, such as affinity and anti-affinity groups. It ensures proper balancing
by grouping guests and evaluating them for further balancing. The class provides methods
to initialize with ProxLB data and to generate groups based on guest and node data.
The groups class is responsible for handling the correlations between the guests
and their groups like affinity and anti-affinity groups. To ensure a proper balancing
guests will ge grouped and then evaluated for further balancing.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
from typing import Dict, Any
from utils.logger import SystemdLogger
from utils.helper import Helper
@@ -22,14 +16,6 @@ class Groups:
The groups class is responsible for handling the correlations between the guests
and their groups like affinity and anti-affinity groups. To ensure a proper balancing
guests will ge grouped and then evaluated for further balancing.
Methods:
__init__(proxlb_data: Dict[str, Any]):
Initializes the Groups class.
get_groups(guests: Dict[str, Any], nodes: Dict[str, Any]) -> Dict[str, Any]:
Generates and returns a dictionary of affinity and anti-affinity groups
based on the provided data.
"""
def __init__(self, proxlb_data: Dict[str, Any]):

View File

@@ -3,16 +3,9 @@ The Guests class retrieves all running guests on the Proxmox cluster across all
It handles both VM and CT guest types, collecting their resource metrics.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
from typing import Dict, Any
from utils.logger import SystemdLogger
from models.pools import Pools
from models.tags import Tags
import time
logger = SystemdLogger()
@@ -21,14 +14,6 @@ class Guests:
"""
The Guests class retrieves all running guests on the Proxmox cluster across all available nodes.
It handles both VM and CT guest types, collecting their resource metrics.
Methods:
__init__:
Initializes the Guests class.
get_guests(proxmox_api: any, nodes: Dict[str, Any]) -> Dict[str, Any]:
Retrieves metrics for all running guests (both VMs and CTs) across all nodes in the Proxmox cluster.
It collects resource metrics such as CPU, memory, and disk usage, as well as tags and affinity/anti-affinity groups.
"""
def __init__(self):
"""
@@ -36,7 +21,7 @@ class Guests:
"""
@staticmethod
def get_guests(proxmox_api: any, pools: Dict[str, Any], nodes: Dict[str, Any], meta: Dict[str, Any], proxlb_config: Dict[str, Any]) -> Dict[str, Any]:
def get_guests(proxmox_api: any, nodes: Dict[str, Any]) -> Dict[str, Any]:
"""
Get metrics of all guests in a Proxmox cluster.
@@ -47,8 +32,6 @@ class Guests:
Args:
proxmox_api (any): The Proxmox API client instance.
nodes (Dict[str, Any]): A dictionary containing information about the nodes in the Proxmox cluster.
meta (Dict[str, Any]): A dictionary containing metadata information.
proxmox_config (Dict[str, Any]): A dictionary containing the ProxLB configuration.
Returns:
Dict[str, Any]: A dictionary containing metrics and information for all running guests.
@@ -67,41 +50,21 @@ class Guests:
if guest['status'] == 'running':
guests['guests'][guest['name']] = {}
guests['guests'][guest['name']]['name'] = guest['name']
guests['guests'][guest['name']]['cpu_total'] = int(guest['cpus'])
guests['guests'][guest['name']]['cpu_used'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', None)
guests['guests'][guest['name']]['cpu_pressure_some_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', 'some')
guests['guests'][guest['name']]['cpu_pressure_full_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', 'full')
guests['guests'][guest['name']]['cpu_pressure_some_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', 'some', spikes=True)
guests['guests'][guest['name']]['cpu_pressure_full_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', 'full', spikes=True)
guests['guests'][guest['name']]['cpu_pressure_hot'] = False
guests['guests'][guest['name']]['cpu_total'] = guest['cpus']
guests['guests'][guest['name']]['cpu_used'] = guest['cpu']
guests['guests'][guest['name']]['memory_total'] = guest['maxmem']
guests['guests'][guest['name']]['memory_used'] = guest['mem']
guests['guests'][guest['name']]['memory_pressure_some_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'memory', 'some')
guests['guests'][guest['name']]['memory_pressure_full_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'memory', 'full')
guests['guests'][guest['name']]['memory_pressure_some_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'memory', 'some', spikes=True)
guests['guests'][guest['name']]['memory_pressure_full_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'memory', 'full', spikes=True)
guests['guests'][guest['name']]['memory_pressure_hot'] = False
guests['guests'][guest['name']]['disk_total'] = guest['maxdisk']
guests['guests'][guest['name']]['disk_used'] = guest['disk']
guests['guests'][guest['name']]['disk_pressure_some_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'disk', 'some')
guests['guests'][guest['name']]['disk_pressure_full_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'disk', 'full')
guests['guests'][guest['name']]['disk_pressure_some_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'disk', 'some', spikes=True)
guests['guests'][guest['name']]['disk_pressure_full_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'disk', 'full', spikes=True)
guests['guests'][guest['name']]['disk_pressure_hot'] = False
guests['guests'][guest['name']]['id'] = guest['vmid']
guests['guests'][guest['name']]['node_current'] = node
guests['guests'][guest['name']]['node_target'] = node
guests['guests'][guest['name']]['processed'] = False
guests['guests'][guest['name']]['pressure_hot'] = False
guests['guests'][guest['name']]['tags'] = Tags.get_tags_from_guests(proxmox_api, node, guest['vmid'], 'vm')
guests['guests'][guest['name']]['pools'] = Pools.get_pools_for_guest(guest['name'], pools)
guests['guests'][guest['name']]['affinity_groups'] = Tags.get_affinity_groups(guests['guests'][guest['name']]['tags'], guests['guests'][guest['name']]['pools'], proxlb_config)
guests['guests'][guest['name']]['anti_affinity_groups'] = Tags.get_anti_affinity_groups(guests['guests'][guest['name']]['tags'], guests['guests'][guest['name']]['pools'], proxlb_config)
guests['guests'][guest['name']]['affinity_groups'] = Tags.get_affinity_groups(guests['guests'][guest['name']]['tags'])
guests['guests'][guest['name']]['anti_affinity_groups'] = Tags.get_anti_affinity_groups(guests['guests'][guest['name']]['tags'])
guests['guests'][guest['name']]['ignore'] = Tags.get_ignore(guests['guests'][guest['name']]['tags'])
guests['guests'][guest['name']]['node_relationships'] = Tags.get_node_relationships(guests['guests'][guest['name']]['tags'], nodes, guests['guests'][guest['name']]['pools'], proxlb_config)
guests['guests'][guest['name']]['type'] = 'vm'
logger.debug(f"Resources of Guest {guest['name']} (type VM) added: {guests['guests'][guest['name']]}")
else:
logger.debug(f'Metric for VM {guest["name"]} ignored because VM is not running.')
@@ -112,97 +75,23 @@ class Guests:
if guest['status'] == 'running':
guests['guests'][guest['name']] = {}
guests['guests'][guest['name']]['name'] = guest['name']
guests['guests'][guest['name']]['cpu_total'] = int(guest['cpus'])
guests['guests'][guest['name']]['cpu_used'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', None)
guests['guests'][guest['name']]['cpu_pressure_some_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', 'some')
guests['guests'][guest['name']]['cpu_pressure_full_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', 'full')
guests['guests'][guest['name']]['cpu_pressure_some_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', 'some', spikes=True)
guests['guests'][guest['name']]['cpu_pressure_full_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'cpu', 'full', spikes=True)
guests['guests'][guest['name']]['cpu_pressure_hot'] = False
guests['guests'][guest['name']]['cpu_total'] = guest['cpus']
guests['guests'][guest['name']]['cpu_used'] = guest['cpu']
guests['guests'][guest['name']]['memory_total'] = guest['maxmem']
guests['guests'][guest['name']]['memory_used'] = guest['mem']
guests['guests'][guest['name']]['memory_pressure_some_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'memory', 'some')
guests['guests'][guest['name']]['memory_pressure_full_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'memory', 'full')
guests['guests'][guest['name']]['memory_pressure_some_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'memory', 'some', spikes=True)
guests['guests'][guest['name']]['memory_pressure_full_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'memory', 'full', spikes=True)
guests['guests'][guest['name']]['memory_pressure_hot'] = False
guests['guests'][guest['name']]['disk_total'] = guest['maxdisk']
guests['guests'][guest['name']]['disk_used'] = guest['disk']
guests['guests'][guest['name']]['disk_pressure_some_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'disk', 'some')
guests['guests'][guest['name']]['disk_pressure_full_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'disk', 'full')
guests['guests'][guest['name']]['disk_pressure_some_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'disk', 'some', spikes=True)
guests['guests'][guest['name']]['disk_pressure_full_spikes_percent'] = Guests.get_guest_rrd_data(proxmox_api, node, guest['vmid'], guest['name'], 'disk', 'full', spikes=True)
guests['guests'][guest['name']]['disk_pressure_hot'] = False
guests['guests'][guest['name']]['id'] = guest['vmid']
guests['guests'][guest['name']]['node_current'] = node
guests['guests'][guest['name']]['node_target'] = node
guests['guests'][guest['name']]['processed'] = False
guests['guests'][guest['name']]['pressure_hot'] = False
guests['guests'][guest['name']]['tags'] = Tags.get_tags_from_guests(proxmox_api, node, guest['vmid'], 'ct')
guests['guests'][guest['name']]['pools'] = Pools.get_pools_for_guest(guest['name'], pools)
guests['guests'][guest['name']]['affinity_groups'] = Tags.get_affinity_groups(guests['guests'][guest['name']]['tags'], guests['guests'][guest['name']]['pools'], proxlb_config)
guests['guests'][guest['name']]['anti_affinity_groups'] = Tags.get_anti_affinity_groups(guests['guests'][guest['name']]['tags'], guests['guests'][guest['name']]['pools'], proxlb_config)
guests['guests'][guest['name']]['affinity_groups'] = Tags.get_affinity_groups(guests['guests'][guest['name']]['tags'])
guests['guests'][guest['name']]['anti_affinity_groups'] = Tags.get_anti_affinity_groups(guests['guests'][guest['name']]['tags'])
guests['guests'][guest['name']]['ignore'] = Tags.get_ignore(guests['guests'][guest['name']]['tags'])
guests['guests'][guest['name']]['node_relationships'] = Tags.get_node_relationships(guests['guests'][guest['name']]['tags'], nodes, guests['guests'][guest['name']]['pools'], proxlb_config)
guests['guests'][guest['name']]['type'] = 'ct'
logger.debug(f"Resources of Guest {guest['name']} (type CT) added: {guests['guests'][guest['name']]}")
else:
logger.debug(f'Metric for CT {guest["name"]} ignored because CT is not running.')
logger.debug("Finished: get_guests.")
return guests
@staticmethod
def get_guest_rrd_data(proxmox_api, node_name: str, vm_id: int, vm_name: str, object_name: str, object_type: str, spikes=False) -> float:
"""
Retrieves the rrd data metrics for a specific resource (CPU, memory, disk) of a guest VM or CT.
Args:
proxmox_api (Any): The Proxmox API client instance.
node_name (str): The name of the node hosting the guest.
vm_id (int): The ID of the guest VM or CT.
vm_name (str): The name of the guest VM or CT.
object_name (str): The resource type to query (e.g., 'cpu', 'memory', 'disk').
object_type (str, optional): The pressure type ('some', 'full') or None for average usage.
spikes (bool, optional): Whether to consider spikes in the calculation. Defaults to False.
Returns:
float: The calculated average usage value for the specified resource.
"""
logger.debug("Starting: get_guest_rrd_data.")
time.sleep(0.1)
try:
if spikes:
logger.debug(f"Getting spike RRD data for {object_name} from guest: {vm_name}.")
guest_data_rrd = proxmox_api.nodes(node_name).qemu(vm_id).rrddata.get(timeframe="hour", cf="MAX")
else:
logger.debug(f"Getting average RRD data for {object_name} from guest: {vm_name}.")
guest_data_rrd = proxmox_api.nodes(node_name).qemu(vm_id).rrddata.get(timeframe="hour", cf="AVERAGE")
except Exception:
logger.error(f"Failed to retrieve RRD data for guest: {vm_name} (ID: {vm_id}) on node: {node_name}. Using 0.0 as value.")
logger.debug("Finished: get_guest_rrd_data.")
return float(0.0)
if object_type:
lookup_key = f"pressure{object_name}{object_type}"
if spikes:
# RRD data is collected every minute, so we look at the last 6 entries
# and take the maximum value to represent the spike
logger.debug(f"Getting RRD data (spike: {spikes}) of pressure for {object_name} {object_type} from guest: {vm_name}.")
rrd_data_value = [row.get(lookup_key) for row in guest_data_rrd if row.get(lookup_key) is not None]
rrd_data_value = max(rrd_data_value[-6:], default=0.0)
else:
# Calculate the average value from the RRD data entries
logger.debug(f"Getting RRD data (spike: {spikes}) of pressure for {object_name} {object_type} from guest: {vm_name}.")
rrd_data_value = sum(entry.get(lookup_key, 0.0) for entry in guest_data_rrd) / len(guest_data_rrd)
else:
logger.debug(f"Getting RRD data of cpu usage from guest: {vm_name}.")
rrd_data_value = sum(entry.get("cpu", 0.0) for entry in guest_data_rrd) / len(guest_data_rrd)
logger.debug(f"RRD data (spike: {spikes}) for {object_name} from guest: {vm_name}: {rrd_data_value}")
logger.debug("Finished: get_guest_rrd_data.")
return rrd_data_value

View File

@@ -1,27 +1,8 @@
"""
The Nodes class retrieves all running nodes in a Proxmox cluster
and collects their resource metrics.
Methods:
__init__:
Initializes the Nodes class.
get_nodes(proxmox_api: any, proxlb_config: Dict[str, Any]) -> Dict[str, Any]:
Gets metrics of all nodes in a Proxmox cluster.
set_node_maintenance(proxlb_config: Dict[str, Any], node_name: str) -> Dict[str, Any]:
Sets Proxmox nodes to a maintenance mode if required.
set_node_ignore(proxlb_config: Dict[str, Any], node_name: str) -> Dict[str, Any]:
Sets Proxmox nodes to be ignored if requested.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import time
from typing import Dict, Any
from utils.logger import SystemdLogger
@@ -48,7 +29,6 @@ class Nodes:
Args:
proxmox_api (any): The Proxmox API client instance.
proxmox_config (Dict[str, Any]): A dictionary containing the ProxLB configuration.
nodes (Dict[str, Any]): A dictionary containing information about the nodes in the Proxmox cluster.
Returns:
@@ -62,21 +42,14 @@ class Nodes:
if node["status"] == "online" and not Nodes.set_node_ignore(proxlb_config, node["node"]):
nodes["nodes"][node["node"]] = {}
nodes["nodes"][node["node"]]["name"] = node["node"]
nodes["nodes"][node["node"]]["pve_version"] = Nodes.get_node_pve_version(proxmox_api, node["node"])
nodes["nodes"][node["node"]]["pressure_hot"] = False
nodes["nodes"][node["node"]]["maintenance"] = False
nodes["nodes"][node["node"]]["cpu_total"] = node["maxcpu"]
nodes["nodes"][node["node"]]["cpu_assigned"] = 0
nodes["nodes"][node["node"]]["cpu_used"] = node["cpu"] * node["maxcpu"]
nodes["nodes"][node["node"]]["cpu_used"] = node["cpu"]
nodes["nodes"][node["node"]]["cpu_free"] = (node["maxcpu"]) - (node["cpu"] * node["maxcpu"])
nodes["nodes"][node["node"]]["cpu_assigned_percent"] = nodes["nodes"][node["node"]]["cpu_assigned"] / nodes["nodes"][node["node"]]["cpu_total"] * 100
nodes["nodes"][node["node"]]["cpu_free_percent"] = nodes["nodes"][node["node"]]["cpu_free"] / node["maxcpu"] * 100
nodes["nodes"][node["node"]]["cpu_used_percent"] = nodes["nodes"][node["node"]]["cpu_used"] / node["maxcpu"] * 100
nodes["nodes"][node["node"]]["cpu_pressure_some_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "cpu", "some")
nodes["nodes"][node["node"]]["cpu_pressure_full_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "cpu", "full")
nodes["nodes"][node["node"]]["cpu_pressure_some_spikes_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "cpu", "some", spikes=True)
nodes["nodes"][node["node"]]["cpu_pressure_full_spikes_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "cpu", "full", spikes=True)
nodes["nodes"][node["node"]]["cpu_pressure_hot"] = False
nodes["nodes"][node["node"]]["memory_total"] = node["maxmem"]
nodes["nodes"][node["node"]]["memory_assigned"] = 0
nodes["nodes"][node["node"]]["memory_used"] = node["mem"]
@@ -84,11 +57,6 @@ class Nodes:
nodes["nodes"][node["node"]]["memory_assigned_percent"] = nodes["nodes"][node["node"]]["memory_assigned"] / nodes["nodes"][node["node"]]["memory_total"] * 100
nodes["nodes"][node["node"]]["memory_free_percent"] = nodes["nodes"][node["node"]]["memory_free"] / node["maxmem"] * 100
nodes["nodes"][node["node"]]["memory_used_percent"] = nodes["nodes"][node["node"]]["memory_used"] / node["maxmem"] * 100
nodes["nodes"][node["node"]]["memory_pressure_some_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "memory", "some")
nodes["nodes"][node["node"]]["memory_pressure_full_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "memory", "full")
nodes["nodes"][node["node"]]["memory_pressure_some_spikes_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "memory", "some", spikes=True)
nodes["nodes"][node["node"]]["memory_pressure_full_spikes_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "memory", "full", spikes=True)
nodes["nodes"][node["node"]]["memory_pressure_hot"] = False
nodes["nodes"][node["node"]]["disk_total"] = node["maxdisk"]
nodes["nodes"][node["node"]]["disk_assigned"] = 0
nodes["nodes"][node["node"]]["disk_used"] = node["disk"]
@@ -96,22 +64,16 @@ class Nodes:
nodes["nodes"][node["node"]]["disk_assigned_percent"] = nodes["nodes"][node["node"]]["disk_assigned"] / nodes["nodes"][node["node"]]["disk_total"] * 100
nodes["nodes"][node["node"]]["disk_free_percent"] = nodes["nodes"][node["node"]]["disk_free"] / node["maxdisk"] * 100
nodes["nodes"][node["node"]]["disk_used_percent"] = nodes["nodes"][node["node"]]["disk_used"] / node["maxdisk"] * 100
nodes["nodes"][node["node"]]["disk_pressure_some_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "disk", "some")
nodes["nodes"][node["node"]]["disk_pressure_full_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "disk", "full")
nodes["nodes"][node["node"]]["disk_pressure_some_spikes_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "disk", "some", spikes=True)
nodes["nodes"][node["node"]]["disk_pressure_full_spikes_percent"] = Nodes.get_node_rrd_data(proxmox_api, node["node"], "disk", "full", spikes=True)
nodes["nodes"][node["node"]]["disk_pressure_hot"] = False
# Evaluate if node should be set to maintenance mode
if Nodes.set_node_maintenance(proxmox_api, proxlb_config, node["node"]):
if Nodes.set_node_maintenance(proxlb_config, node["node"]):
nodes["nodes"][node["node"]]["maintenance"] = True
logger.debug(f"Node metrics collected: {nodes}")
logger.debug("Finished: get_nodes.")
return nodes
@staticmethod
def set_node_maintenance(proxmox_api, proxlb_config: Dict[str, Any], node_name: str) -> Dict[str, Any]:
def set_node_maintenance(proxlb_config: Dict[str, Any], node_name: str) -> Dict[str, Any]:
"""
Set nodes to maintenance mode based on the provided configuration.
@@ -119,7 +81,6 @@ class Nodes:
based on the configuration provided in proxlb_config.
Args:
proxmox_api (any): The Proxmox API client instance.
proxlb_config (Dict[str, Any]): A dictionary containing the ProxLB configuration, including maintenance nodes.
node_name: (str): The current node name within the outer iteration.
@@ -128,24 +89,11 @@ class Nodes:
"""
logger.debug("Starting: set_node_maintenance.")
# Evaluate maintenance mode by config
if proxlb_config.get("proxmox_cluster", None).get("maintenance_nodes", None) is not None:
if len(proxlb_config.get("proxmox_cluster", {}).get("maintenance_nodes", [])) > 0:
if node_name in proxlb_config.get("proxmox_cluster", {}).get("maintenance_nodes", []):
logger.info(f"Node: {node_name} has been set to maintenance mode (by ProxLB config).")
logger.warning(f"Node: {node_name} has been set to maintenance mode.")
return True
else:
logger.debug(f"Node: {node_name} is not in maintenance mode by ProxLB config.")
# Evaluate maintenance mode by Proxmox HA
for ha_element in proxmox_api.cluster.ha.status.current.get():
if ha_element.get("status"):
if "maintenance mode" in ha_element.get("status"):
if ha_element.get("node") == node_name:
logger.info(f"Node: {node_name} has been set to maintenance mode (by Proxmox HA API).")
return True
else:
logger.debug(f"Node: {node_name} is not in maintenance mode by Proxmox HA API.")
logger.debug("Finished: set_node_maintenance.")
@@ -169,87 +117,7 @@ class Nodes:
if proxlb_config.get("proxmox_cluster", None).get("ignore_nodes", None) is not None:
if len(proxlb_config.get("proxmox_cluster", {}).get("ignore_nodes", [])) > 0:
if node_name in proxlb_config.get("proxmox_cluster", {}).get("ignore_nodes", []):
logger.info(f"Node: {node_name} has been set to be ignored. Not adding node!")
logger.warning(f"Node: {node_name} has been set to be ignored. Not adding node!")
return True
logger.debug("Finished: set_node_ignore.")
@staticmethod
def get_node_rrd_data(proxmox_api, node_name: str, object_name: str, object_type: str, spikes=False) -> float:
"""
Retrieves the rrd data metrics for a specific resource (CPU, memory, disk) of a node.
Args:
proxmox_api (Any): The Proxmox API client instance.
node_name (str): The name of the node hosting the guest.
object_name (str): The resource type to query (e.g., 'cpu', 'memory', 'disk').
object_type (str, optional): The pressure type ('some', 'full') or None for average usage.
spikes (bool, optional): Whether to consider spikes in the calculation. Defaults to False.
Returns:
float: The calculated average usage value for the specified resource.
"""
logger.debug("Starting: get_node_rrd_data.")
time.sleep(0.1)
try:
if spikes:
logger.debug(f"Getting spike RRD data for {object_name} from node: {node_name}.")
node_data_rrd = proxmox_api.nodes(node_name).rrddata.get(timeframe="hour", cf="MAX")
else:
logger.debug(f"Getting average RRD data for {object_name} from node: {node_name}.")
node_data_rrd = proxmox_api.nodes(node_name).rrddata.get(timeframe="hour", cf="AVERAGE")
except Exception:
logger.error(f"Failed to retrieve RRD data for guest: {node_name}. Using 0.0 as value.")
logger.debug("Finished: get_node_rrd_data.")
return 0.0
lookup_key = f"pressure{object_name}{object_type}"
if spikes:
# RRD data is collected every minute, so we look at the last 6 entries
# and take the maximum value to represent the spike
rrd_data_value = [row.get(lookup_key) for row in node_data_rrd if row.get(lookup_key) is not None]
rrd_data_value = max(rrd_data_value[-6:], default=0.0)
else:
# Calculate the average value from the RRD data entries
rrd_data_value = sum(entry.get(lookup_key, 0.0) for entry in node_data_rrd) / len(node_data_rrd)
logger.debug(f"RRD data (spike: {spikes}) for {object_name} from node: {node_name}: {rrd_data_value}")
logger.debug("Finished: get_node_rrd_data.")
return rrd_data_value
@staticmethod
def get_node_pve_version(proxmox_api, node_name: str) -> float:
"""
Return the Proxmox VE (PVE) version for a given node by querying the Proxmox API.
This function calls proxmox_api.nodes(node_name).version.get() and extracts the
'version' field from the returned mapping. The value is expected to be numeric
(or convertible to float) and is returned as a float.
Args:
proxmox_api (Any): The Proxmox API client instance.
node_name (str): The name of the node hosting the guest.
Returns:
float: The PVE version for the specified node as a floating point number.
Raises:
Exception: If the proxmox_api call fails, returns an unexpected structure, or the
'version' field is missing or cannot be converted to float. Callers should
handle or propagate exceptions as appropriate.
"""
logger.debug("Starting: get_node_pve_version.")
time.sleep(0.1)
try:
logger.debug(f"Trying to get PVE version for node: {node_name}.")
version = proxmox_api.nodes(node_name).version.get()
except Exception:
logger.error(f"Failed to get PVE version for node: {node_name}.")
logger.debug(f"Got version {version['version']} for node {node_name}.")
logger.debug("Finished: get_node_pve_version.")
return version["version"]

View File

@@ -1,117 +0,0 @@
"""
The Pools class retrieves all present pools defined on a Proxmox cluster
including the chield objects.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
from typing import Dict, Any
from utils.logger import SystemdLogger
from models.tags import Tags
import time
logger = SystemdLogger()
class Pools:
"""
The Pools class retrieves all present pools defined on a Proxmox cluster
including the chield objects.
Methods:
__init__:
Initializes the Pools class.
get_pools(proxmox_api: any) -> Dict[str, Any]:
Retrieve pool definitions and membership from the Proxmox cluster.
Returns a dict with a top-level "pools" mapping each poolid to
{"name": <poolid>, "members": [<member_names>...]}.
This method does not collect per-member metrics or perform node filtering.
"""
def __init__(self):
"""
Initializes the Pools class with the provided ProxLB data.
"""
@staticmethod
def get_pools(proxmox_api: any) -> Dict[str, Any]:
"""
Retrieve all pools and their members from a Proxmox cluster.
Queries the Proxmox API for pool definitions and returns a dictionary
containing each pool's id/name and a list of its member VM/CT names.
This function does not perform per-member metric collection or node
filtering — it only gathers pool membership information.
Args:
proxmox_api (any): Proxmox API client instance.
Returns:
Dict[str, Any]: Dictionary with a top-level "pools" key mapping poolid
to {"name": <poolid>, "members": [<member_names>...]}.
"""
logger.debug("Starting: get_pools.")
pools = {"pools": {}}
# Pool objects: iterate over all pools in the cluster.
# We keep pool members even if their nodes are ignored so resource accounting
# for rebalancing remains correct and we avoid overprovisioning nodes.
for pool in proxmox_api.pools.get():
logger.debug(f"Got pool: {pool['poolid']}")
pools['pools'][pool['poolid']] = {}
pools['pools'][pool['poolid']]['name'] = pool['poolid']
pools['pools'][pool['poolid']]['members'] = []
# Fetch pool details and collect member names
pool_details = proxmox_api.pools(pool['poolid']).get()
for member in pool_details.get("members", []):
# We might also have objects without the key "name", e.g. storage pools
if "name" not in member:
logger.debug(f"Skipping member without name in pool: {pool['poolid']}")
continue
logger.debug(f"Got member: {member['name']} for pool: {pool['poolid']}")
pools['pools'][pool['poolid']]['members'].append(member["name"])
logger.debug("Finished: get_pools.")
return pools
@staticmethod
def get_pools_for_guest(guest_name: str, pools: Dict[str, Any]) -> Dict[str, Any]:
"""
Return the list of pool names that include the given guest.
Args:
guest_name (str): Name of the VM or CT to look up.
pools (Dict[str, Any]): Pools structure as returned by get_pools(),
expected to contain a top-level "pools" mapping each poolid to
{"name": <poolid>, "members": [<member_names>...]}.
Returns:
list[str]: Names of pools the guest is a member of (empty list if none).
"""
logger.debug("Starting: get_pools_for_guests.")
guest_pools = []
for pool in pools.items():
for pool_id, pool_data in pool[1].items():
if type(pool_data) is dict:
pool_name = pool_data.get("name", "")
pool_name_members = pool_data.get("members", [])
if guest_name in pool_name_members:
logger.debug(f"Guest: {guest_name} is member of Pool: {pool_name}.")
guest_pools.append(pool_name)
else:
logger.debug(f"Guest: {guest_name} is NOT member of Pool: {pool_name}.")
else:
logger.debug(f"Pool data for pool_id {pool_id} is not a dict: {pool_data}")
logger.debug("Finished: get_pools_for_guests.")
return guest_pools

View File

@@ -1,50 +1,25 @@
"""
The Tags class retrieves and processes tags from guests of type VM or CT running
in a Proxmox cluster. It provides methods to fetch tags from the Proxmox API and
evaluate them for affinity, anti-affinity, and ignore tags, which are used during
balancing calculations.
The Tags class retrieves all tags from guests of the type VM or CT running
in a Proxmox cluster and validates for affinity, anti-affinity and ignore
tags set for the guest in the Proxmox API.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import time
from typing import List
from typing import Dict, Any
from utils.logger import SystemdLogger
from utils.helper import Helper
logger = SystemdLogger()
class Tags:
"""
The Tags class retrieves and processes tags from guests of type VM or CT running
in a Proxmox cluster. It provides methods to fetch tags from the Proxmox API and
evaluate them for affinity, anti-affinity, and ignore tags, which are used during
balancing calculations.
Methods:
__init__:
Initializes the Tags class.
get_tags_from_guests(proxmox_api: any, node: str, guest_id: int, guest_type: str) -> List[str]:
Retrieves all tags for a given guest from the Proxmox API.
get_affinity_groups(tags: List[str]) -> List[str]:
Evaluates and returns all affinity tags from the provided list of tags.
get_anti_affinity_groups(tags: List[str]) -> List[str]:
Evaluates and returns all anti-affinity tags from the provided list of tags.
get_ignore(tags: List[str]) -> bool:
Evaluates and returns a boolean indicating whether the guest should be ignored based on the provided list of tags.
The Tags class retrieves all tags from guests of the type VM or CT running
in a Proxmox cluster and validates for affinity, anti-affinity and ignore
tags set for the guest in the Proxmox API.
"""
def __init__(self):
"""
Initializes the tags class.
Initializes the Tags class.
"""
@staticmethod
@@ -80,18 +55,15 @@ class Tags:
return tags
@staticmethod
def get_affinity_groups(tags: List[str], pools: List[str], proxlb_config: Dict[str, Any]) -> List[str]:
def get_affinity_groups(tags: List[str]) -> List[str]:
"""
Get affinity tags for a guest from the Proxmox cluster by the API.
This method retrieves all tags for a given guest or based on a
membership of a pool and evaluates the affinity groups which are
required during the balancing calculations.
This method retrieves all tags for a given guest and evaluates the
affinity tags which are required during the balancing calculations.
Args:
tags (List): A list holding all defined tags for a given guest.
pools (List): A list holding all defined pools for a given guest.
proxlb_config (Dict): A dict holding the ProxLB configuration.
Returns:
List: A list including all affinity tags for the given guest.
@@ -102,36 +74,21 @@ class Tags:
if len(tags) > 0:
for tag in tags:
if tag.startswith("plb_affinity"):
logger.debug(f"Adding affinity group for tag {tag}.")
affinity_tags.append(tag)
else:
logger.debug(f"Skipping affinity group for tag {tag}.")
if len(pools) > 0:
for pool in pools:
if pool in (proxlb_config['balancing'].get('pools') or {}):
if proxlb_config['balancing']['pools'][pool].get('type', None) == 'affinity':
logger.debug(f"Adding affinity group for pool {pool}.")
affinity_tags.append(pool)
else:
logger.debug(f"Skipping affinity group for pool {pool}.")
logger.debug("Finished: get_affinity_groups.")
return affinity_tags
@staticmethod
def get_anti_affinity_groups(tags: List[str], pools: List[str], proxlb_config: Dict[str, Any]) -> List[str]:
def get_anti_affinity_groups(tags: List[str]) -> List[str]:
"""
Get anti-affinity tags for a guest from the Proxmox cluster by the API.
This method retrieves all tags for a given guest or based on a
membership of a pool and evaluates the anti-affinity groups which
are required during the balancing calculations.
This method retrieves all tags for a given guest and evaluates the
anti-affinity tags which are required during the balancing calculations.
Args:
tags (List): A list holding all defined tags for a given guest.
pools (List): A list holding all defined pools for a given guest.
proxlb_config (Dict): A dict holding the ProxLB configuration.
Returns:
List: A list including all anti-affinity tags for the given guest..
@@ -142,19 +99,7 @@ class Tags:
if len(tags) > 0:
for tag in tags:
if tag.startswith("plb_anti_affinity"):
logger.debug(f"Adding anti-affinity group for tag {tag}.")
anti_affinity_tags.append(tag)
else:
logger.debug(f"Skipping anti-affinity group for tag {tag}.")
if len(pools) > 0:
for pool in pools:
if pool in (proxlb_config['balancing'].get('pools') or {}):
if proxlb_config['balancing']['pools'][pool].get('type', None) == 'anti-affinity':
logger.debug(f"Adding anti-affinity group for pool {pool}.")
anti_affinity_tags.append(pool)
else:
logger.debug(f"Skipping anti-affinity group for pool {pool}.")
logger.debug("Finished: get_anti_affinity_groups.")
return anti_affinity_tags
@@ -171,7 +116,7 @@ class Tags:
tags (List): A list holding all defined tags for a given guest.
Returns:
Bool: Returns a bool that indicates whether to ignore a guest or not.
Bool: Returns a bool that indicates wether to ignore a guest or not.
"""
logger.debug("Starting: get_ignore.")
ignore_tag = False
@@ -183,57 +128,3 @@ class Tags:
logger.debug("Finished: get_ignore.")
return ignore_tag
@staticmethod
def get_node_relationships(tags: List[str], nodes: Dict[str, Any], pools: List[str], proxlb_config: Dict[str, Any]) -> str:
"""
Get a node relationship tag for a guest from the Proxmox cluster by the API to pin
a guest to a node or by defined pools from ProxLB configuration.
This method retrieves a relationship tag between a guest and a specific
hypervisor node to pin the guest to a specific node (e.g., for licensing reason).
Args:
tags (List): A list holding all defined tags for a given guest.
nodes (Dict): A dictionary holding all available nodes in the cluster.
pools (List): A list holding all defined pools for a given guest.
proxlb_config (Dict): A dict holding the ProxLB configuration.
Returns:
Str: The related hypervisor node name(s).
"""
logger.debug("Starting: get_node_relationships.")
node_relationship_tags = []
if len(tags) > 0:
logger.debug("Validating node pinning by tags.")
for tag in tags:
if tag.startswith("plb_pin"):
node_relationship_tag = tag.replace("plb_pin_", "")
# Validate if the node to pin is present in the cluster
if Helper.validate_node_presence(node_relationship_tag, nodes):
logger.debug(f"Tag {node_relationship_tag} is valid! Defined node exists in the cluster.")
logger.debug(f"Setting node relationship because of tag {tag} to {node_relationship_tag}.")
node_relationship_tags.append(node_relationship_tag)
else:
logger.warning(f"Tag {node_relationship_tag} is invalid! Defined node does not exist in the cluster. Not applying pinning.")
if len(pools) > 0:
logger.debug("Validating node pinning by pools.")
for pool in pools:
if pool in (proxlb_config['balancing'].get('pools') or {}):
node = proxlb_config['balancing']['pools'][pool].get('pin', None)
# Validate if the node to pin is present in the cluster
if Helper.validate_node_presence(node, nodes):
logger.debug(f"Pool pinning tag {node} is valid! Defined node exists in the cluster.")
logger.debug(f"Setting node relationship because of pool {pool} to {node}.")
node_relationship_tags.append(node)
else:
logger.warning(f"Pool pinning tag {node} is invalid! Defined node does not exist in the cluster. Not applying pinning.")
else:
logger.debug(f"Skipping pinning for pool {pool}. Pool is not defined in ProxLB configuration.")
logger.debug("Finished: get_node_relationships.")
return node_relationship_tags

View File

@@ -2,11 +2,6 @@
The CliParser class handles the parsing of command-line interface (CLI) arguments.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import argparse
import utils.version
from utils.logger import SystemdLogger

View File

@@ -3,11 +3,6 @@ The ConfigParser class handles the parsing of configuration file
from a given YAML file from any location.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import os
import sys
try:
@@ -29,16 +24,8 @@ logger = SystemdLogger()
class ConfigParser:
"""
The ConfigParser class handles the parsing of a configuration file.
Methods:
__init__(config_path: str)
test_config_path(config_path: str) -> None
Checks if the configuration file is present at the given config path.
get_config() -> Dict[str, Any]
Parses and returns the configuration data from the YAML file.
The ConfigParser class handles the parsing of configuration file
from a given YAML file from any location.
"""
def __init__(self, config_path: str):
"""

View File

@@ -3,21 +3,12 @@ The Helper class provides some basic helper functions to not mess up the code in
classes.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import json
import uuid
import re
import socket
import sys
import time
import utils.version
from utils.logger import SystemdLogger
from typing import Dict, Any
from types import FrameType
logger = SystemdLogger()
@@ -26,25 +17,7 @@ class Helper:
"""
The Helper class provides some basic helper functions to not mess up the code in other
classes.
Methods:
__init__():
Initializes the general Helper class.
get_uuid_string() -> str:
Generates a random uuid and returns it as a string.
log_node_metrics(proxlb_data: Dict[str, Any], init: bool = True) -> None:
Logs the memory, CPU, and disk usage metrics of nodes in the provided proxlb_data dictionary.
get_version(print_version: bool = False) -> None:
Returns the current version of ProxLB and optionally prints it to stdout.
get_daemon_mode(proxlb_config: Dict[str, Any]) -> None:
Checks if the daemon mode is active and handles the scheduling accordingly.
"""
proxlb_reload = False
def __init__(self):
"""
Initializes the general Helper clas.
@@ -121,215 +94,12 @@ class Helper:
None
"""
logger.debug("Starting: get_daemon_mode.")
if proxlb_config.get("service", {}).get("daemon", True):
# Validate schedule format which changed in v1.1.1
if type(proxlb_config["service"].get("schedule", None)) != dict:
logger.error("Invalid format for schedule. Please use 'hours' or 'minutes'.")
sys.exit(1)
# Convert hours to seconds
if proxlb_config["service"]["schedule"].get("format", "hours") == "hours":
sleep_seconds = proxlb_config.get("service", {}).get("schedule", {}).get("interval", 12) * 3600
# Convert minutes to seconds
elif proxlb_config["service"]["schedule"].get("format", "hours") == "minutes":
sleep_seconds = proxlb_config.get("service", {}).get("schedule", {}).get("interval", 720) * 60
else:
logger.error("Invalid format for schedule. Please use 'hours' or 'minutes'.")
sys.exit(1)
logger.info(f"Daemon mode active: Next run in: {proxlb_config.get('service', {}).get('schedule', {}).get('interval', 12)} {proxlb_config['service']['schedule'].get('format', 'hours')}.")
if proxlb_config.get("service", {}).get("daemon", False):
sleep_seconds = proxlb_config.get("service", {}).get("schedule", 12) * 3600
logger.info(f"Daemon mode active: Next run in: {proxlb_config.get('service', {}).get('schedule', 12)} hours.")
time.sleep(sleep_seconds)
else:
logger.debug("Successfully executed ProxLB. Daemon mode not active - stopping.")
print("Daemon mode not active - stopping.")
logger.debug("Daemon mode is not active.")
sys.exit(0)
logger.debug("Finished: get_daemon_mode.")
@staticmethod
def get_service_delay(proxlb_config: Dict[str, Any]) -> None:
"""
Checks if a start up delay for the service is defined and waits to proceed until
the time is up.
Parameters:
proxlb_config (Dict[str, Any]): A dictionary containing the ProxLB configuration.
Returns:
None
"""
logger.debug("Starting: get_service_delay.")
if proxlb_config.get("service", {}).get("delay", {}).get("enable", False):
# Convert hours to seconds
if proxlb_config["service"]["delay"].get("format", "hours") == "hours":
sleep_seconds = proxlb_config.get("service", {}).get("delay", {}).get("time", 1) * 3600
# Convert minutes to seconds
elif proxlb_config["service"]["delay"].get("format", "hours") == "minutes":
sleep_seconds = proxlb_config.get("service", {}).get("delay", {}).get("time", 60) * 60
else:
logger.error("Invalid format for service delay. Please use 'hours' or 'minutes'.")
sys.exit(1)
logger.info(f"Service delay active: First run in: {proxlb_config.get('service', {}).get('delay', {}).get('time', 1)} {proxlb_config['service']['delay'].get('format', 'hours')}.")
time.sleep(sleep_seconds)
else:
logger.debug("Service delay not active. Proceeding without delay.")
logger.debug("Finished: get_service_delay.")
@staticmethod
def print_json(proxlb_config: Dict[str, Any], print_json: bool = False) -> None:
"""
Prints the calculated balancing matrix as a JSON output to stdout.
Parameters:
proxlb_config (Dict[str, Any]): A dictionary containing the ProxLB configuration.
Returns:
None
"""
logger.debug("Starting: print_json.")
if print_json:
# Create a filtered list by stripping the 'meta' key from the proxlb_config dictionary
# to make sure that no credentials are leaked.
filtered_data = {k: v for k, v in proxlb_config.items() if k != "meta"}
print(json.dumps(filtered_data, indent=4))
logger.debug("Finished: print_json.")
@staticmethod
def handler_sighup(signum: int, frame: FrameType) -> None:
"""
Signal handler for SIGHUP.
This method is triggered when the process receives a SIGHUP signal.
It sets the `proxlb_reload` class variable to True to indicate that
configuration should be reloaded in the main loop.
Args:
signum (int): The signal number (expected to be signal.SIGHUP).
frame (frame object): Current stack frame (unused but required by signal handler signature).
"""
logger.debug("Starting: handle_sighup.")
logger.debug("Got SIGHUP signal. Reloading...")
Helper.proxlb_reload = True
logger.debug("Finished: handle_sighup.")
@staticmethod
def handler_sigint(signum: int, frame: FrameType) -> None:
"""
Signal handler for SIGINT. (triggered by CTRL+C).
Args:
signum (int): The signal number (e.g., SIGINT).
frame (FrameType): The current stack frame when the signal was received.
Returns:
None
"""
exit_message = "ProxLB has been successfully terminated by user."
logger.debug(exit_message)
print(f"\n {exit_message}")
sys.exit(0)
@staticmethod
def get_host_port_from_string(host_object):
"""
Parses a string containing a host (IPv4, IPv6, or hostname) and an optional port, and returns a tuple of (host, port).
Supported formats:
- Hostname or IPv4 without port: "example.com" or "192.168.0.1"
- Hostname or IPv4 with port: "example.com:8006" or "192.168.0.1:8006"
- IPv6 in brackets with optional port: "[fc00::1]" or "[fc00::1]:8006"
- IPv6 without brackets, port is assumed after last colon: "fc00::1:8006"
If no port is specified, port 8006 is used as the default.
Args:
host_object (str): A string representing a host with or without a port.
Returns:
tuple: A tuple (host: str, port: int)
"""
logger.debug("Starting: get_host_port_from_string.")
# IPv6 (with or without port, written in brackets)
match = re.match(r'^\[(.+)\](?::(\d+))?$', host_object)
if match:
host = match.group(1)
port = int(match.group(2)) if match.group(2) else 8006
return host, port
# Count colons to identify IPv6 addresses without brackets
colon_count = host_object.count(':')
# IPv4 or hostname without port
if colon_count == 0:
return host_object, 8006
# IPv4 or hostname with port
elif colon_count == 1:
host, port = host_object.split(':')
return host, int(port)
# IPv6 (with or without port, assume last colon is port)
else:
parts = host_object.rsplit(':', 1)
try:
port = int(parts[1])
return parts[0], port
except ValueError:
return host_object, 8006
@staticmethod
def validate_node_presence(node: str, nodes: Dict[str, Any]) -> bool:
"""
Validates whether a given node exists in the provided cluster nodes dictionary.
Args:
node (str): The name of the node to validate.
nodes (Dict[str, Any]): A dictionary containing cluster information.
Must include a "nodes" key mapping to a dict of available nodes.
Returns:
bool: True if the node exists in the cluster, False otherwise.
"""
logger.debug("Starting: validate_node_presence.")
if node in nodes["nodes"].keys():
logger.info(f"Node {node} found in cluster. Applying pinning.")
logger.debug("Finished: validate_node_presence.")
return True
else:
logger.warning(f"Node {node} not found in cluster. Not applying pinning!")
logger.debug("Finished: validate_node_presence.")
return False
@staticmethod
def tcp_connect_test(addr_family: int, host: str, port: int, timeout: int) -> tuple[bool, int | None]:
"""
Attempt a TCP connection to the specified host and port to test the reachability.
Args:
addr_family (int): Address family for the socket (e.g., socket.AF_INET for IPv4, socket.AF_INET6 for IPv6).
host (str): The hostname or IP address to connect to.
port (int): The port number to connect to.
timeout (int): Connection timeout in seconds.
Returns:
tuple[bool, int | None]: A tuple containing:
- bool: True if the connection was successful, False otherwise.
- int | None: None if the connection was successful, otherwise the errno code indicating the reason for failure.
"""
test_socket = socket.socket(addr_family, socket.SOCK_STREAM)
test_socket.settimeout(timeout)
try:
rc = test_socket.connect_ex((host, port))
return (rc == 0, rc if rc != 0 else None)
finally:
test_socket.close()

View File

@@ -1,15 +1,10 @@
"""
The SystemdLogger class provides a singleton logger that integrates with systemd's journal if available.
It dynamically evaluates the environment and adjusts the logger accordingly.
The SystemdLogger class provides the root logger support. It dynamically
evaluates the further usage and imports of journald and adjusts
the logger to the systems functionality where it gets executed
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import logging
import sys
try:
from systemd.journal import JournalHandler
SYSTEMD_PRESENT = True
@@ -19,36 +14,9 @@ except ImportError:
class SystemdLogger:
"""
The SystemdLogger class provides a singleton logger that integrates with systemd's journal if available.
It dynamically evaluates the environment and adjusts the logger accordingly.
Attributes:
instance (SystemdLogger): Singleton instance of the SystemdLogger class.
Methods:
__new__(cls, name: str = "ProxLB", level: str = logging.INFO) -> 'SystemdLogger':
Creates a new instance of the SystemdLogger class or returns the existing instance.
initialize_logger(self, name: str, level: str) -> None:
Initializes the logger with the given name and log level. Adds a JournalHandler if systemd is present.
set_log_level(self, level: str) -> None:
Sets the log level for the logger and all its handlers.
debug(self, msg: str) -> str:
Logs a message with level DEBUG.
info(self, msg: str) -> str:
Logs a message with level INFO.
warning(self, msg: str) -> str:
Logs a message with level WARNING.
error(self, msg: str) -> str:
Logs a message with level ERROR.
critical(self, msg: str) -> str:
Logs a message with level CRITICAL.
The SystemdLogger class provides the root logger support. It dynamically
evaluates the further usage and imports of journald and adjusts
the logger to the systems functionality where it gets executed.
"""
# Create a singleton instance variable
instance = None
@@ -83,22 +51,17 @@ class SystemdLogger:
self.logger = logging.getLogger(name)
self.logger.setLevel(level)
# Create a logging handler depending on the
# capabilities of the underlying OS where systemd
# logging is preferred.
# Create a JournalHandler for systemd integration if this
# is supported on the underlying OS.
if SYSTEMD_PRESENT:
# Add a JournalHandler for systemd integration
handler = JournalHandler(SYSLOG_IDENTIFIER="ProxLB")
else:
# Add a stdout handler as a fallback
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(level)
# Set a formatter to include the logger's name and log message
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
# Add handler to logger
self.logger.addHandler(handler)
journal_handler = JournalHandler()
journal_handler.setLevel(level)
# Set a formatter to include the logger's name and log message
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
journal_handler.setFormatter(formatter)
# Add handler to logger
self.logger.addHandler(journal_handler)
def set_log_level(self, level: str) -> None:
"""

View File

@@ -1,19 +1,7 @@
"""
The proxmox_api class manages connections to the Proxmox API by parsing the required objects
for the authentication which can be based on username/password or API tokens.
This class provides methods to initialize the Proxmox API connection, test connectivity to
Proxmox hosts, and handle authentication using either username/password or API tokens.
It also includes functionality to distribute load across multiple Proxmox API endpoints
and manage SSL certificate validation.
Module providing a function printing python version.
"""
__author__ = "Florian Paul Azim Hoberg <gyptazy>"
__copyright__ = "Copyright (C) 2025 Florian Paul Azim Hoberg (@gyptazy)"
__license__ = "GPL-3.0"
import errno
try:
import proxmoxer
PROXMOXER_PRESENT = True
@@ -21,20 +9,18 @@ except ImportError:
PROXMOXER_PRESENT = False
import random
import socket
import sys
try:
import requests
REQUESTS_PRESENT = True
except ImportError:
REQUESTS_PRESENT = False
import sys
import time
try:
import urllib3
URLLIB3_PRESENT = True
except ImportError:
URLLIB3_PRESENT = False
from typing import Dict, Any
from utils.helper import Helper
from utils.logger import SystemdLogger
@@ -56,103 +42,32 @@ logger = SystemdLogger()
class ProxmoxApi:
"""
The proxmox_api class manages connections to the Proxmox API by parsing the required objects
for the authentication which can be based on username/password or API tokens.
This class provides methods to initialize the Proxmox API connection, test connectivity to
Proxmox hosts, and handle authentication using either username/password or API tokens.
It also includes functionality to distribute load across multiple Proxmox API endpoints
and manage SSL certificate validation.
Attributes:
logger (SystemdLogger): Logger instance for logging messages.
proxmox_api (proxmoxer.ProxmoxAPI): Authenticated ProxmoxAPI object.
Methods:
__init__(proxlb_config: Dict[str, Any]) -> None:
Initializes the ProxmoxApi instance with the provided configuration.
__getattr__(name):
Delegates attribute access to the proxmox_api object.
api_connect_get_hosts(proxmox_api_endpoints: list) -> str:
Determines a working Proxmox API host from a list of endpoints.
test_api_proxmox_host(host: str) -> str:
Tests connectivity to a Proxmox host by resolving its IP address.
test_api_proxmox_host_ipv4(host: str, port: int = 8006, timeout: int = 1) -> bool:
Tests reachability of a Proxmox host on its IPv4 address.
test_api_proxmox_host_ipv6(host: str, port: int = 8006, timeout: int = 1) -> bool:
Tests reachability of a Proxmox host on its IPv6 address.
api_connect(proxlb_config: Dict[str, Any]) -> proxmoxer.ProxmoxAPI:
Establishes a connection to the Proxmox API using the provided configuration.
Handles command-line argument parsing for ProxLB.
"""
def __init__(self, proxlb_config: Dict[str, Any]) -> None:
"""
Initializes the ProxmoxApi instance with the provided configuration.
Initialize the ProxmoxApi instance.
This constructor method sets up the Proxmox API connection by calling the
api_connect method with the given configuration dictionary. It logs the
initialization process for debugging purposes.
This method sets up the ProxmoxApi instance by testing the required module dependencies
and establishing a connection to the Proxmox API using the provided configuration.
Args:
proxlb_config (Dict[str, Any]): A dictionary containing the Proxmox API configuration.
proxlb_config (Dict[str, Any]): Configuration dictionary containing Proxmox API connection details.
Returns:
None
"""
logger.debug("Starting: ProxmoxApi initialization.")
self.proxmox_api = self.api_connect(proxlb_config)
self.test_api_user_permissions(self.proxmox_api)
logger.debug("Finished: ProxmoxApi initialization.")
def __getattr__(self, name):
"""
Delegate attribute access to proxmox_api to the underlying proxmoxer module.
Delegate attribute access to proxmox_api.
"""
return getattr(self.proxmox_api, name)
def validate_config(self, proxlb_config: Dict[str, Any]) -> None:
"""
Validates the provided ProxLB configuration dictionary to ensure that it contains
the necessary credentials for authentication and that the credentials are not
mutually exclusive.
Args:
proxlb_config (Dict[str, Any]): A dictionary containing the ProxLB configuration.
It must include a "proxmox_api" key with a nested dictionary that contains
either "user" and "password" keys for username/password authentication or
"token_id" and "token_secret" keys for API token authentication.
Raises:
SystemExit: If both pass/token_secret and API token authentication methods are
provided, the function will log a critical error message and terminate
the program.
Logs:
Logs the start and end of the validation process. Logs a critical error if both
authentication methods are provided.
"""
logger.debug("Starting: validate_config.")
if not proxlb_config.get("proxmox_api", False):
logger.critical(f"Config error. Please check your proxmox_api chapter in your config file.")
print(f"Config error. Please check your proxmox_api chapter in your config file.")
sys.exit(1)
proxlb_credentials = proxlb_config["proxmox_api"]
present_auth_pass = "pass" in proxlb_credentials
present_auth_secret = "token_secret" in proxlb_credentials
token_id = proxlb_credentials.get("token_id", None)
if token_id:
non_allowed_chars = ["@", "!"]
for char in non_allowed_chars:
if char in token_id:
logger.error(f"Wrong user/token format defined. User and token id must be splitted! Please see: https://github.com/gyptazy/ProxLB/blob/main/docs/03_configuration.md#required-permissions-for-a-user")
sys.exit(1)
if present_auth_pass and present_auth_secret:
logger.critical(f"Username/password and API token authentication are mutal exclusive. Please use only one!")
print(f"Username/password and API token authentication are mutal exclusive. Please use only one!")
sys.exit(1)
logger.debug("Finished: validate_config.")
def api_connect_get_hosts(self, proxlb_config, proxmox_api_endpoints: list) -> str:
def api_connect_get_hosts(self, proxmox_api_endpoints: list) -> str:
"""
Perform a connectivity test to determine a working host for the Proxmox API.
@@ -163,7 +78,6 @@ class ProxmoxApi:
are found, one is chosen at random to distribute the load across the cluster.
Args:
proxlb_config (Dict[str, Any]): A dictionary containing the ProxLB configuration.
proxmox_api_endpoints (list): A list of Proxmox API endpoints to test.
Returns:
@@ -176,40 +90,45 @@ class ProxmoxApi:
logger.debug("Starting: api_connect_get_hosts.")
# Pre-validate the given API endpoints
if not isinstance(proxmox_api_endpoints, list):
logger.critical("The proxmox_api hosts are not defined as a list type.")
logger.critical(f"The proxmox_api hosts are not defined as a list type.")
sys.exit(1)
if not proxmox_api_endpoints:
logger.critical("No proxmox_api hosts are defined.")
logger.critical(f"No proxmox_api hosts are defined.")
sys.exit(1)
validated_api_hosts: list[tuple[str, int]] = []
if len(proxmox_api_endpoints) == 0:
logger.critical(f"No proxmox_api hosts are defined.")
sys.exit(1)
# Get a suitable Proxmox API endpoint. Therefore, we check if we only have
# a single Proxmox API endpoint or multiple ones. If only one, we can return
# this one immediately. If this one does not work, the urllib will raise an
# exception during the connection attempt.
if len(proxmox_api_endpoints) == 1:
return proxmox_api_endpoints[0]
# If we have multiple Proxmox API endpoints, we need to check each one by
# doing a connection attempt for IPv4 and IPv6. If we find a working one,
# we return that one. This allows us to define multiple endpoints in a cluster.
validated_api_hosts = []
for host in proxmox_api_endpoints:
retries = proxlb_config["proxmox_api"].get("retries", 1)
wait_time = proxlb_config["proxmox_api"].get("wait_time", 1)
validated = self.test_api_proxmox_host(host)
if validated:
validated_api_hosts.append(validated)
for attempt in range(retries):
candidate_host, candidate_port = self.test_api_proxmox_host(host)
if candidate_host:
validated_api_hosts.append((candidate_host, candidate_port))
break
else:
logger.warning(
f"Attempt {attempt + 1}/{retries} failed for host {host}. "
f"Retrying in {wait_time} seconds..."
)
time.sleep(wait_time)
if validated_api_hosts:
chosen_host, chosen_port = random.choice(validated_api_hosts)
return chosen_host, chosen_port
if len(validated_api_hosts) > 0:
# Choose a random host to distribute the load across the cluster
# as a simple load balancing mechanism.
return random.choice(validated_api_hosts)
logger.critical("No valid Proxmox API hosts found.")
print("No valid Proxmox API hosts found.")
logger.debug("Finished: api_connect_get_hosts.")
sys.exit(1)
def test_api_proxmox_host(self, host: str) -> tuple[str, int | None, None]:
def test_api_proxmox_host(self, host: str) -> str:
"""
Tests the connectivity to a Proxmox host by resolving its IP address and
checking both IPv4 and IPv6 addresses.
@@ -228,37 +147,20 @@ class ProxmoxApi:
bool: False if the Proxmox server is not reachable.
"""
logger.debug("Starting: test_api_proxmox_host.")
ip = socket.getaddrinfo(host, None, socket.AF_UNSPEC)
for address_type in ip:
if address_type[0] == socket.AF_INET:
logger.debug(f"{host} is type ipv4.")
if self.test_api_proxmox_host_ipv4(host):
return host
elif address_type[0] == socket.AF_INET6:
logger.debug(f"{host} is type ipv6.")
if self.test_api_proxmox_host_ipv6(host):
return host
else:
return False
# Validate for custom port configurations (e.g., by given external
# loadbalancer systems)
host, port = Helper.get_host_port_from_string(host)
if port is None:
port = 8006
# Try resolving DNS to IP and log non-resolvable ones
try:
infos = socket.getaddrinfo(host, None, socket.AF_UNSPEC)
except socket.gaierror:
logger.warning(f"Could not resolve {host}.")
return (None, None)
# Check both families that are actually present
saw_family = set()
for family, *_rest in infos:
saw_family.add(family)
if socket.AF_INET in saw_family:
logger.debug(f"{host} has IPv4.")
if self.test_api_proxmox_host_ipv4(host, port):
return (host, port)
if socket.AF_INET6 in saw_family:
logger.debug(f"{host} has IPv6.")
if self.test_api_proxmox_host_ipv6(host, port):
return (host, port)
logger.debug("Finished: test_api_proxmox_host (unreachable).")
return (None, None)
logger.debug("Finished: test_api_proxmox_host.")
def test_api_proxmox_host_ipv4(self, host: str, port: int = 8006, timeout: int = 1) -> bool:
"""
@@ -277,16 +179,18 @@ class ProxmoxApi:
bool: True if the host is reachable on the specified port, False otherwise.
"""
logger.debug("Starting: test_api_proxmox_host_ipv4.")
ok, rc = Helper.tcp_connect_test(socket.AF_INET, host, port, timeout)
if ok:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(timeout)
logger.warning(f"Warning: Host {host} ran into a timout when connectoing on IPv4 for tcp/{port}.")
result = sock.connect_ex((host, port))
if result == 0:
sock.close()
logger.debug(f"Host {host} is reachable on IPv4 for tcp/{port}.")
logger.debug("Finished: test_api_proxmox_host_ipv4.")
return True
if rc == errno.ETIMEDOUT:
logger.warning(f"Timeout connecting to {host} on IPv4 tcp/{port}.")
else:
logger.warning(f"Host {host} is unreachable on IPv4 for tcp/{port} (errno {rc}).")
sock.close()
logger.warning(f"Host {host} is unreachable on IPv4 for tcp/{port}.")
logger.debug("Finished: test_api_proxmox_host_ipv4.")
return False
@@ -308,58 +212,22 @@ class ProxmoxApi:
bool: True if the host is reachable on the specified port, False otherwise.
"""
logger.debug("Starting: test_api_proxmox_host_ipv6.")
ok, rc = Helper.tcp_connect_test(socket.AF_INET6, host, port, timeout)
if ok:
sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
sock.settimeout(timeout)
logger.warning(f"Host {host} ran into a timout when connectoing on IPv6 for tcp/{port}.")
result = sock.connect_ex((host, port))
if result == 0:
sock.close()
logger.debug(f"Host {host} is reachable on IPv6 for tcp/{port}.")
logger.debug("Finished: test_api_proxmox_host_ipv6.")
return True
if rc == errno.ETIMEDOUT:
logger.warning(f"Timeout connecting to {host} on IPv6 tcp/{port}.")
else:
logger.warning(f"Host {host} is unreachable on IPv6 for tcp/{port} (errno {rc}).")
sock.close()
logger.warning(f"Host {host} is unreachable on IPv6 for tcp/{port}.")
logger.debug("Finished: test_api_proxmox_host_ipv6.")
logger.debug("Finished: test_api_proxmox_host_ipv4.")
return False
def test_api_user_permissions(self, proxmox_api: any):
"""
Test the permissions of the current user/token used for the Proxmox API.
This method gets all assigned permissions for all API paths for the current
used user/token and validates them against the minimum required permissions.
Args:
proxmox_api (any): The Proxmox API client instance.
"""
logger.debug("Starting: test_api_user_permissions.")
permissions_required = ["Datastore.Audit", "Sys.Audit", "VM.Audit", "VM.Migrate"]
permissions_available = []
# Get the permissions for the current user/token from API
try:
permissions = proxmox_api.access.permissions.get()
except proxmoxer.core.ResourceException as api_error:
if "no such user" in str(api_error):
logger.error("Authentication to Proxmox API not possible: User not known - please check your username and config file.")
sys.exit(1)
else:
logger.error(f"Proxmox API error: {api_error}")
sys.exit(1)
# Get all available permissions of the current user/token
for path, permission in permissions.items():
for permission in permissions[path]:
permissions_available.append(permission)
# Validate if all required permissions are included within the available permissions
for required_permission in permissions_required:
if required_permission not in permissions_available:
logger.critical(f"Permission '{required_permission}' is missing. Please adjust the permissions for your user/token. See also: https://github.com/gyptazy/ProxLB/blob/main/docs/03_configuration.md#required-permissions-for-a-user")
sys.exit(1)
logger.debug("Finished: test_api_user_permissions.")
def api_connect(self, proxlb_config: Dict[str, Any]) -> proxmoxer.ProxmoxAPI:
"""
Establishes a connection to the Proxmox API using the provided configuration.
@@ -388,11 +256,8 @@ class ProxmoxApi:
requests.exceptions.ConnectionError: If the connection to the Proxmox API is refused.
"""
logger.debug("Starting: api_connect.")
# Validate config
self.validate_config(proxlb_config)
# Get a valid Proxmox API endpoint
proxmox_api_endpoint, proxmox_api_port = self.api_connect_get_hosts(proxlb_config, proxlb_config.get("proxmox_api", {}).get("hosts", []))
proxmox_api_endpoint = self.api_connect_get_hosts(proxlb_config.get("proxmox_api", {}).get("hosts", []))
# Disable warnings for SSL certificate validation
if not proxlb_config.get("proxmox_api").get("ssl_verification", True):
@@ -402,26 +267,12 @@ class ProxmoxApi:
# Login into Proxmox API and create API object
try:
if proxlb_config.get("proxmox_api").get("token_secret", False):
proxmox_api = proxmoxer.ProxmoxAPI(
proxmox_api_endpoint,
port=proxmox_api_port,
user=proxlb_config.get("proxmox_api").get("user", True),
token_name=proxlb_config.get("proxmox_api").get("token_id", True),
token_value=proxlb_config.get("proxmox_api").get("token_secret", True),
verify_ssl=proxlb_config.get("proxmox_api").get("ssl_verification", True),
timeout=proxlb_config.get("proxmox_api").get("timeout", True))
logger.debug("Using API token authentication.")
else:
proxmox_api = proxmoxer.ProxmoxAPI(
proxmox_api_endpoint,
port=proxmox_api_port,
user=proxlb_config.get("proxmox_api").get("user", True),
password=proxlb_config.get("proxmox_api").get("pass", True),
verify_ssl=proxlb_config.get("proxmox_api").get("ssl_verification", True),
timeout=proxlb_config.get("proxmox_api").get("timeout", True))
logger.debug("Using username/password authentication.")
proxmox_api = proxmoxer.ProxmoxAPI(
proxmox_api_endpoint,
user=proxlb_config.get("proxmox_api").get("user", True),
password=proxlb_config.get("proxmox_api").get("pass", True),
verify_ssl=proxlb_config.get("proxmox_api").get("ssl_verification", True),
timeout=proxlb_config.get("proxmox_api").get("timeout", True))
except proxmoxer.backends.https.AuthenticationError as proxmox_api_error:
logger.critical(f"Authentication failed. Please check the defined credentials: {proxmox_api_error}")
sys.exit(2)
@@ -436,5 +287,6 @@ class ProxmoxApi:
sys.exit(2)
logger.info(f"API connection to host {proxmox_api_endpoint} succeeded.")
logger.debug("Finished: api_connect.")
return proxmox_api

Some files were not shown because too many files have changed in this diff Show More