Update issue templates

release: Create stable release 1.0.4

.changelogs/1.0.3/64_improve_error_handling_and_validations.yml

@@ -0,0 +1,2 @@
+fixed:
+  - Improved the overall validation and error handling. [#64]

.changelogs/1.0.3/67_fix_anti_affinity_rules.yml

@@ -0,0 +1,2 @@
+fixed:
+  - Fix anti-affinity rules not evaluating a new and different node. [#67]

.changelogs/1.0.3/68_adjust_logging_ha_services.yml

@@ -0,0 +1,2 @@
+changed:
+  - Provide a more reasonable output when HA services are not active in a Proxmox cluster. [#68]

.changelogs/1.0.3/71_fix_ignore_vm_node_handling_if_unset.yml

@@ -0,0 +1,2 @@
+fixed:
+  - Fix handling of unset `ignore_nodes` and `ignore_vms` resulted in an attribute error. [#71]

.changelogs/1.0.3/74_fix_documentation_master_only.yml

@@ -0,0 +1,2 @@
+fixed:
+  - Fix documentation for the master_only parameter placed in the wrong config section. [#74]

.changelogs/1.0.3/docs_fix_default_config_add_dedicated_user_howto.yml

@@ -0,0 +1,4 @@
+fixed:
+  - Fix bug in the `proxlb.conf` in the vm_balancing section.
+added:
+  - Add doc how to add dedicated user for authentication. (by @Dulux-Oz) 

.changelogs/1.0.3/release_meta.yml

@@ -1 +1 @@
-date: TBD
+date: 2024-09-12

.changelogs/1.0.4/58_add_maintenance_mode.yml

@@ -0,0 +1,2 @@
+added:
+  - Add maintenance mode to evacuate a node and move workloads for other nodes in the cluster. [#58]

.changelogs/1.0.4/75_fix_cpu_balancing.yml

@@ -0,0 +1,2 @@
+fixed:
+  - Fix CPU balancing where calculations are done in float instead of int. (by @glitchvern) [#75]

.changelogs/1.0.4/79_storage_balancing_whitelist.yml

@@ -0,0 +1,3 @@
+changed:
+  - Run storage balancing only on supported shared storages. [#79]
+  - Run storage balancing only when needed to save time. [#79]

.changelogs/1.0.4/81_documentation_infrastrucutre.yml

@@ -0,0 +1,2 @@
+fixed:
+  - Fix documentation for the underlying infrastructure. [#81]

.changelogs/1.0.4/89_add_version_output.yml

@@ -0,0 +1,2 @@
+added:
+  - Add version output cli arg. [#89]

.changelogs/1.0.4/91_make_api_timeout_configureable.yml

@@ -0,0 +1,2 @@
+added:
+  - Add feature to make API timeout configureable. [#91]

.changelogs/1.0.4/release_meta.yml

@@ -0,0 +1 @@
+date: 2024-10-11

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,27 @@
+---
+name: Bug report
+about: Create a bug report
+title: "`Bug`:"
+labels: bug, needs-analysis
+assignees: ''
+
+---
+
+## General
+<-- Describe the bug from a high level perspective. -->
+
+## Weighting
+Score: <-- Define a scoring from 0-10 (10 highest, most urgent) -->
+
+## Config
+<-- Attach the ProxLB configuration for further analysis. Please take car to NOT publish your API credentials! -->
+
+## Log
+<-- Attach the ProxLB debug log for further analysis. Please take car to NOT publish your API credentials! -->
+
+## Meta
+Please provide some more information about your setup. This includes where you obtained ProxLB (e.g., as a `.deb` file, from the repository or container image) and also which version you're running in which mode. You can obtain the used version from you image version, your local repository information or by running `proxlb -v`.
+
+Version: <-- DEFINE_VERSION -->
+Installed from: <-- DEFINE_INSTALL_SOURCE -->
+Running as: <-- Container, local on Proxmox, local on all Proxmox, dedicated -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,14 @@
+---
+name: Feature request
+about: Create a new request for a missing feature
+title: "`Feature`: "
+labels: feature, needs-analysis
+assignees: ''
+
+---
+
+## General
+<-- Describe the feature idea from a high level perspective. -->
+
+## Details
+<-- Provide some more details about the new feature request and provide examples. -->

CHANGELOG.md

@@ -6,34 +6,82 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
+## [1.0.4] - 2024-10-11
+
+### Added
+
+- Add feature to make API timeout configureable. [#91]
+- Add maintenance mode to evacuate a node and move workloads for other nodes in the cluster. [#58]
+- Add version output cli arg. [#89]
+
+### Changed
+
+- Run storage balancing only on supported shared storages. [#79]
+- Run storage balancing only when needed to save time. [#79]
+
+### Fixed
+
+- Fix CPU balancing where calculations are done in float instead of int. (by @glitchvern) [#75]
+- Fix documentation for the underlying infrastructure. [#81]
+
+
+## [1.0.3] - 2024-09-12
+
+### Added
+
+- Add storage balancing function. [#51]
+- Add a convert function to cast all bool alike options from configparser to bools. [#53]
+- Add a config parser options for future features. [#53]
+- Add a config versio schema that must be supported by ProxLB. [#53]
+- Add doc how to add dedicated user for authentication. (by @Dulux-Oz)
+- Add feature to allow the API hosts being provided as a comma separated list. [#60]
+- Add cli arg `-b` to return the next best node for next VM/CT placement. [#8]
+
+### Changed
+
+- Improve the underlying code base for future implementations. [#53]
+- Provide a more reasonable output when HA services are not active in a Proxmox cluster. [#68]
+
+### Fixed
+
+- Fixed `master_only` function by inverting the condition.
+- Improved the overall validation and error handling. [#64]
+- Fix bug in the `proxlb.conf` in the vm_balancing section.
+- Fix anti-affinity rules not evaluating a new and different node. [#67]
+- Fix documentation for the master_only parameter placed in the wrong config section. [#74]
+- Fix handling of unset `ignore_nodes` and `ignore_vms` resulted in an attribute error. [#71]
+
+
 ## [1.0.2] - 2024-08-13
 
 ### Added
 
-- Add option to run migration in parallel or sequentially. [#41]
 - Add option to run ProxLB only on the Proxmox's master node in the cluster (reg. HA feature). [#40]
+- Add option to run migrations in parallel or sequentially. [#41]
 
 ### Changed
 
 - Fix daemon timer to use hours instead of minutes. [#45]
+
+### Fixed
+
 - Fix CMake packaging for Debian package to avoid overwriting the config file. [#49]
-- Fix wonkey code style.
 
 
 ## [1.0.0] - 2024-08-01
 
 ### Added
 
-- Add feature to prevent VMs from being relocated by defining a wildcard pattern. [#7]
-- Add feature to make log verbosity configurable [#17].
 - Add option_mode to rebalance by node's free resources in percent (instead of bytes). [#29]
-- Add option to rebalance by assigned VM resources to avoid over provisioning. [#16]
-- Add Docker/Podman support. [#10 by @daanbosch]
-- Add exclude grouping feature to rebalance VMs from being located together to new nodes. [#4]
-- Add feature to prevent VMs from being relocated by defining the 'plb_ignore_vm' tag. [#7]
-- Add dry-run support to see what kind of rebalancing would be done. [#6]
 - Add LXC/Container integration. [#27]
+- Add exclude grouping feature to rebalance VMs from being located together to new nodes. [#4]
+- Add dry-run support to see what kind of rebalancing would be done. [#6]
+- Add Docker/Podman support. [#10 by @daanbosch]
+- Add feature to prevent VMs from being relocated by defining a wildcard pattern. [#7]
+- Add feature to prevent VMs from being relocated by defining the 'plb_ignore_vm' tag. [#7]
 - Add include grouping feature to rebalance VMs bundled to new nodes. [#3]
+- Add option to rebalance by assigned VM resources to avoid overprovisioning. [#16]
+- Add feature to make log verbosity configurable [#17].
 
 ### Changed
 

README.md

@@ -1,5 +1,5 @@
 # ProxLB - (Re)Balance VM Workloads in Proxmox Clusters
-<img align="left" src="https://cdn.gyptazy.ch/images/Prox-LB-logo.jpg"/>
+<img align="left" src="https://cdn.gyptazy.com/images/Prox-LB-logo.jpg"/>
 <br>
 
 <p float="center"><img src="https://img.shields.io/github/license/gyptazy/ProxLB"/><img src="https://img.shields.io/github/contributors/gyptazy/ProxLB"/><img src="https://img.shields.io/github/last-commit/gyptazy/ProxLB/main"/><img src="https://img.shields.io/github/issues-raw/gyptazy/ProxLB"/><img src="https://img.shields.io/github/issues-pr/gyptazy/ProxLB"/></p>
@@ -15,6 +15,7 @@
   - [Usage](#usage)
     - [Dependencies](#dependencies)
     - [Options](#options)
+      - [Notes](#notes)
     - [Parameters](#parameters)
     - [Balancing](#balancing)
       - [General](#general)
@@ -36,10 +37,13 @@
   - [Downloads](#downloads)
     - [Packages](#packages)
     - [Repository](#repository)
+      - [Stable Releases](#stable-releases)
+      - [Beta/Testing Releases](#betatesting-releases)
     - [Container Images (Docker/Podman)](#container-images-dockerpodman)
   - [Misc](#misc)
     - [Bugs](#bugs)
     - [Contributing](#contributing)
+    - [Documentation](#documentation)
     - [Support](#support)
     - [Author(s)](#authors)
 
@@ -53,7 +57,7 @@ Intelligent rebalancing is a key feature of ProxLB where it re-balances VMs base
 Automated rebalancing reduces the need for manual actions, allowing operators to focus on other critical tasks, thereby increasing operational efficiency.
 
 ### Video of Migration
-<img src="https://cdn.gyptazy.ch/images/proxlb-rebalancing-demo.gif"/>
+<img src="https://cdn.gyptazy.com/images/proxlb-rebalancing-demo.gif"/>
 
 ## Features
 * Rebalance VMs/CTs in the cluster by:
@@ -106,10 +110,11 @@ The following options can be set in the `proxlb.conf` file:
 
 | Section | Option | Example | Description |
 |------|:------:|:------:|:------:|
-| `proxmox` | api_host | hypervisor01.gyptazy.ch | Host or IP address (or comma separated list) of the remote Proxmox API. |
+| `proxmox` | api_host | hypervisor01.gyptazy.com | Host or IP address (or comma separated list) of the remote Proxmox API. |
 | | api_user | root@pam | Username for the API. |
 | | api_pass | FooBar | Password for the API. |
 | | verify_ssl | 1 | Validate SSL certificates (1) or ignore (0). (default: 1) |
+| | timeout | 10 | Timeout for the Proxmox API in sec. (default: 10) |
 | `vm_balancing` | enable | 1 | Enables VM/CT balancing. |
 | | method | memory | Defines the balancing method (default: memory) where you can use `memory`, `disk` or `cpu`. |
 | | mode | used | Rebalance by `used` resources (efficiency) or `assigned` (avoid overprovisioning) resources. (default: used)|
@@ -117,9 +122,9 @@ The following options can be set in the `proxlb.conf` file:
 | | type | vm | Rebalance only `vm` (virtual machines), `ct` (containers) or `all` (virtual machines & containers). (default: vm)|
 | | balanciness | 10 | Value of the percentage of lowest and highest resource consumption on nodes may differ before rebalancing. (default: 10) |
 | | parallel_migrations | 1 | Defines if migrations should be done parallely or sequentially. (default: 1) |
+| | maintenance_nodes | dummynode03,dummynode04 | Defines a comma separated list of nodes to set them into maintenance mode and move VMs/CTs to other nodes. |
 | | ignore_nodes | dummynode01,dummynode02,test* | Defines a comma separated list of nodes to exclude. |
 | | ignore_vms | testvm01,testvm02 | Defines a comma separated list of VMs to exclude. (`*` as suffix wildcard or tags are also supported) |
-| | master_only | 0 | Defines is this should only be performed (1) on the cluster master node or not (0). (default: 0) |
 | `storage_balancing` | enable | 0 | Enables storage balancing. |
 | | balanciness | 10 | Value of the percentage of lowest and highest storage consumption may differ before rebalancing. (default: 10) |
 | | parallel_migrations | 1 | Defines if migrations should be done parallely or sequentially. (default: 1) |
@@ -127,16 +132,18 @@ The following options can be set in the `proxlb.conf` file:
 | `api` | enable | 0 | Enables the ProxLB API. |
 | `service`| daemon | 1 | Run as a daemon (1) or one-shot (0). (default: 1) |
 | | schedule | 24 | Hours to rebalance in hours. (default: 24) |
+| | master_only | 0 | Defines is this should only be performed (1) on the cluster master node or not (0). (default: 0) |
 | | log_verbosity | INFO | Defines the log level (default: CRITICAL) where you can use `INFO`, `WARN` or `CRITICAL` |
 | | config_version | 3 | Defines the current config version schema for ProxLB |
 
 An example of the configuration file looks like:
 ```
 [proxmox]
-api_host: hypervisor01.gyptazy.ch
+api_host: hypervisor01.gyptazy.com
 api_user: root@pam
 api_pass: FooBar
 verify_ssl: 1
+timeout: 10
 [vm_balancing]
 enable: 1
 method: memory
@@ -152,6 +159,7 @@ balanciness: 10
 # Enable parallel migrations. If set to 0 it will wait for completed migrations
 # before starting next migration.
 parallel_migrations: 1
+maintenance_nodes: dummynode03,dummynode04
 ignore_nodes: dummynode01,dummynode02
 ignore_vms: testvm01,testvm02
 [storage_balancing]
@@ -161,7 +169,7 @@ enable: 0
 [api]
 enable: 0
 [service]
-# The master_only option might be usuful if running ProxLB on all nodes in a cluster
+# The master_only option might be useful if running ProxLB on all nodes in a cluster
 # but only a single one should do the balancing. The master node is obtained from the Proxmox
 # HA status.
 master_only: 0
@@ -169,6 +177,19 @@ daemon: 1
 config_version: 3
 ```
 
+#### Notes
+* If running ProxLB on more than one Proxmox node you can set `api_host` to a comma-separated list of each node's IP address or hostname. (Example: `api_host: node01.gyptazy.com,node02.gyptazy.com,node03.gyptazy.com`)
+* The `verify_ssl` parameter can switch between the mode to verify trusted remote certificates. Keep in mind, that even local ones are **not** trusted by default and need to be imported to the truststore.
+* Even when using only the `vm_balancing` mode, ensure to have the other sections listed in your config:
+```
+[storage_balancing]
+enable: 0
+[update_service]
+enable: 0
+[api]
+enable: 0
+```
+
 ### Parameters
 The following options and parameters are currently supported:
 
@@ -178,6 +199,8 @@ The following options and parameters are currently supported:
 | -d | --dry-run | Performs a dry-run without doing any actions. | Unset |
 | -j | --json | Returns a JSON of the VM movement. | Unset |
 | -b | --best-node | Returns the best next node for a VM/CT placement (useful for further usage with Terraform/Ansible). | Unset |
+| -m | --maintenance | Sets node(s) to maintenance mode & moves workloads away. | Unset |
+| -v | --version | Returns the ProxLB version on stdout. | Unset |
 
 ### Balancing
 #### General
@@ -216,13 +239,13 @@ Limitations:
 
 ### Affinity Rules / Grouping Relationships
 #### Affinity (Stay Together)
-<img align="left" src="https://cdn.gyptazy.ch/images/plb-rebalancing-include-balance-group.jpg"/> Access the Proxmox Web UI by opening your web browser and navigating to your Proxmox VE web interface, then log in with your credentials. Navigate to the VM you want to tag by selecting it from the left-hand navigation panel. Click on the "Options" tab to view the VM's options, then select "Edit" or "Add" (depending on whether you are editing an existing tag or adding a new one). In the tag field, enter plb_include_ followed by your unique identifier, for example, plb_include_group1. Save the changes to apply the tag to the VM. Repeat these steps for each VM that should be included in the group.
+<img align="left" src="https://cdn.gyptazy.com/images/plb-rebalancing-include-balance-group.jpg"/> Access the Proxmox Web UI by opening your web browser and navigating to your Proxmox VE web interface, then log in with your credentials. Navigate to the VM you want to tag by selecting it from the left-hand navigation panel. Click on the "Options" tab to view the VM's options, then select "Edit" or "Add" (depending on whether you are editing an existing tag or adding a new one). In the tag field, enter plb_include_ followed by your unique identifier, for example, plb_include_group1. Save the changes to apply the tag to the VM. Repeat these steps for each VM that should be included in the group.
 
 #### Anti-Affinity (Keep Apart)
-<img align="left" src="https://cdn.gyptazy.ch/images/plb-rebalancing-exclude-balance-group.jpg"/> Access the Proxmox Web UI by opening your web browser and navigating to your Proxmox VE web interface, then log in with your credentials. Navigate to the VM you want to tag by selecting it from the left-hand navigation panel. Click on the "Options" tab to view the VM's options, then select "Edit" or "Add" (depending on whether you are editing an existing tag or adding a new one). In the tag field, enter plb_exclude_ followed by your unique identifier, for example, plb_exclude_critical. Save the changes to apply the tag to the VM. Repeat these steps for each VM that should be excluded from being on the same node.
+<img align="left" src="https://cdn.gyptazy.com/images/plb-rebalancing-exclude-balance-group.jpg"/> Access the Proxmox Web UI by opening your web browser and navigating to your Proxmox VE web interface, then log in with your credentials. Navigate to the VM you want to tag by selecting it from the left-hand navigation panel. Click on the "Options" tab to view the VM's options, then select "Edit" or "Add" (depending on whether you are editing an existing tag or adding a new one). In the tag field, enter plb_exclude_ followed by your unique identifier, for example, plb_exclude_critical. Save the changes to apply the tag to the VM. Repeat these steps for each VM that should be excluded from being on the same node.
 
 #### Ignore VMs (Tag Style)
-<img align="left" src="https://cdn.gyptazy.ch/images/plb-rebalancing-ignore-vm.jpg"/>  In Proxmox, you can ensure that certain VMs are ignored during the rebalancing process by setting a specific tag within the Proxmox Web UI, rather than solely relying on configurations in the ProxLB config file. This can be achieved by adding the tag 'plb_ignore_vm' to the VM. Once this tag is applied, the VM will be excluded from any further rebalancing operations, simplifying the management process.
+<img align="left" src="https://cdn.gyptazy.com/images/plb-rebalancing-ignore-vm.jpg"/>  In Proxmox, you can ensure that certain VMs are ignored during the rebalancing process by setting a specific tag within the Proxmox Web UI, rather than solely relying on configurations in the ProxLB config file. This can be achieved by adding the tag 'plb_ignore_vm' to the VM. Once this tag is applied, the VM will be excluded from any further rebalancing operations, simplifying the management process.
 
 ### Systemd
 When installing a Linux distribution (such as .deb or .rpm) file, this will be shipped with a systemd unit file. The default configuration file will be sourced from `/etc/proxlb/proxlb.conf`.
@@ -239,7 +262,7 @@ A manual installation is possible and also supports BSD based systems. Proxmox R
 The executable must be able to read the config file, if no dedicated config file is given by the `-c` argument, PLB tries to read it from `/etc/proxlb/proxlb.conf`.
 
 ### Proxmox GUI Integration
-<img align="left" src="https://cdn.gyptazy.ch/images/proxlb-GUI-integration.jpg"/> PLB can also be directly be used from the Proxmox Web UI by installing the optional package `pve-proxmoxlb-service-ui` package which has a dependency on the `proxlb` package. For the Web UI integration, it requires to be installed (in addition) on the nodes on the cluster. Afterwards, a new menu item is present in the HA chapter called `Rebalancing`. This chapter provides two possibilities:
+<img align="left" src="https://cdn.gyptazy.com/images/proxlb-GUI-integration.jpg"/> PLB can also be directly be used from the Proxmox Web UI by installing the optional package `pve-proxmoxlb-service-ui` package which has a dependency on the `proxlb` package. For the Web UI integration, it requires to be installed (in addition) on the nodes on the cluster. Afterwards, a new menu item is present in the HA chapter called `Rebalancing`. This chapter provides two possibilities:
 * Rebalancing VM workloads
 * Migrate VM workloads away from a defined node (e.g. maintenance preparation)
 
@@ -247,8 +270,8 @@ The executable must be able to read the config file, if no dedicated config file
 The easiest way to get started is by using the ready-to-use packages that I provide on my CDN and to run it on a Linux Debian based system. This can also be one of the Proxmox nodes itself.
 
 ```
-wget https://cdn.gyptazy.ch/files/amd64/debian/proxlb/proxlb_1.0.2_amd64.deb
-dpkg -i proxlb_1.0.2_amd64.deb
+wget https://cdn.gyptazy.com/files/os/debian/proxlb/proxlb_1.0.4_amd64.deb
+dpkg -i proxlb_1.0.4_amd64.deb
 # Adjust your config
 vi /etc/proxlb/proxlb.conf
 systemctl restart proxlb
@@ -301,48 +324,55 @@ Here you can find some overviews of references for and about the ProxLB (PLB):
 
 | Description | Link |
 |------|:------:|
-| General introduction into ProxLB | https://gyptazy.ch/blog/proxlb-rebalancing-vm-workloads-across-nodes-in-proxmox-clusters/ |
-| Howto install and use ProxLB on Debian to rebalance vm workloads in a Proxmox cluster | https://gyptazy.ch/howtos/howto-install-and-use-proxlb-to-rebalance-vm-workloads-across-nodes-in-proxmox-clusters/ |
+| General introduction into ProxLB | https://gyptazy.com/blog/proxlb-rebalancing-vm-workloads-across-nodes-in-proxmox-clusters/ |
+| Howto install and use ProxLB on Debian to rebalance vm workloads in a Proxmox cluster | https://gyptazy.com/howtos/howto-install-and-use-proxlb-to-rebalance-vm-workloads-across-nodes-in-proxmox-clusters/ |
 
 ## Downloads
 ProxLB can be obtained in man different ways, depending on which use case you prefer. You can use simply copy the code from GitHub, use created packages for Debian or RedHat based systems, use a Repository to keep ProxLB always up to date or simply use a Container image for Docker/Podman.
 
 ### Packages
 Ready to use packages can be found at:
-* https://cdn.gyptazy.ch/files/amd64/debian/proxlb/
-* https://cdn.gyptazy.ch/files/amd64/ubuntu/proxlb/
-* https://cdn.gyptazy.ch/files/amd64/redhat/proxlb/
-* https://cdn.gyptazy.ch/files/amd64/freebsd/proxlb/
-
+* https://cdn.gyptazy.com/files/os/debian/proxlb/
+* https://cdn.gyptazy.com/files/os/ubuntu/proxlb/
+* https://cdn.gyptazy.com/files/os/redhat/proxlb/
 
 ### Repository
 Debian based systems can also use the repository by adding the following line to their apt sources:
 
+#### Stable Releases
 ```
-deb https://repo.gyptazy.ch/ /
+deb https://repo.gyptazy.com/stable /
 ```
 
-The Repository's GPG key can be found at: `https://repo.gyptazy.ch/repo/KEY.gpg`
+#### Beta/Testing Releases
+```
+deb https://repo.gyptazy.com/testing /
+```
+
+The Repository's GPG key can be found at: `https://repo.gyptazy.com/repository.gpg`
 
 You can also simply import it by running:
 
 ```
-# KeyID:  DEB76ADF7A0BAADB51792782FD6A7A70C11226AA
-# SHA256: 5e44fffa09c747886ee37cc6e9e7eaf37c6734443cc648eaf0a9241a89084383  KEY.gpg
+# KeyID:  17169F23F9F71A14AD49EDADDB51D3EB01824F4C
+# UID:    gyptazy Solutions Repository <contact@gyptazy.com>
+# SHA256: 52c267e6f4ec799d40cdbdb29fa518533ac7942dab557fa4c217a76f90d6b0f3  repository.gpg
 
-wget -O /etc/apt/trusted.gpg.d/proxlb.asc https://repo.gyptazy.ch/repo/KEY.gpg
+wget -O /etc/apt/trusted.gpg.d/proxlb.asc https://repo.gyptazy.com/repository.gpg
 ```
 
-*Note: The defined repositories `repo.gyptazy.ch` and `repo.proxlb.de` are the same!*
+*Note: The defined repositories `repo.gyptazy.com` and `repo.proxlb.de` are the same!*
 
 ### Container Images (Docker/Podman)
 Container Images for Podman, Docker etc., can be found at:
 | Version | Image |
 |------|:------:|
-| latest | cr.gyptazy.ch/proxlb/proxlb:latest |
-| v1.0.2 | cr.gyptazy.ch/proxlb/proxlb:v1.0.2 |
-| v1.0.0 | cr.gyptazy.ch/proxlb/proxlb:v1.0.0 |
-| v0.9.9 | cr.gyptazy.ch/proxlb/proxlb:v0.9.9 |
+| latest | cr.gyptazy.com/proxlb/proxlb:latest |
+| v1.0.4 | cr.gyptazy.com/proxlb/proxlb:v1.0.4 |
+| v1.0.3 | cr.gyptazy.com/proxlb/proxlb:v1.0.3 |
+| v1.0.2 | cr.gyptazy.com/proxlb/proxlb:v1.0.2 |
+| v1.0.0 | cr.gyptazy.com/proxlb/proxlb:v1.0.0 |
+| v0.9.9 | cr.gyptazy.com/proxlb/proxlb:v0.9.9 |
 
 ## Misc
 ### Bugs
@@ -351,14 +381,17 @@ Bugs can be reported via the GitHub issue tracker [here](https://github.com/gypt
 ### Contributing
 Feel free to add further documentation, to adjust already existing one or to contribute with code. Please take care about the style guide and naming conventions. You can find more in our [CONTRIBUTING.md](https://github.com/gyptazy/ProxLB/blob/main/CONTRIBUTING.md) file.
 
+### Documentation
+You can also find additional and more detailed documentation within the [docs/](https://github.com/gyptazy/ProxLB/tree/main/docs) directory.
+
 ### Support
-If you need assistance or have any questions, we offer support through our dedicated [chat room](https://matrix.to/#/#proxlb:gyptazy.ch) in Matrix and on Reddit. Join our community for real-time help, advice, and discussions. Connect with us in our dedicated chat room for immediate support and live interaction with other users and developers. You can also visit our [GitHub Community](https://github.com/gyptazy/ProxLB/discussions/) to post your queries, share your experiences, and get support from fellow community members and moderators. You may also just open directly an issue [here](https://github.com/gyptazy/ProxLB/issues) on GitHub. We are here to help and ensure you have the best experience possible.
+If you need assistance or have any questions, we offer support through our dedicated [chat room](https://matrix.to/#/#proxlb:gyptazy.com) in Matrix and on Reddit. Join our community for real-time help, advice, and discussions. Connect with us in our dedicated chat room for immediate support and live interaction with other users and developers. You can also visit our [GitHub Community](https://github.com/gyptazy/ProxLB/discussions/) to post your queries, share your experiences, and get support from fellow community members and moderators. You may also just open directly an issue [here](https://github.com/gyptazy/ProxLB/issues) on GitHub. We are here to help and ensure you have the best experience possible.
 
 | Support Channel | Link |
 |------|:------:|
-| Matrix | [#proxlb:gyptazy.ch](https://matrix.to/#/#proxlb:gyptazy.ch) |
+| Matrix | [#proxlb:gyptazy.com](https://matrix.to/#/#proxlb:gyptazy.com) |
 | GitHub Community | [GitHub Community](https://github.com/gyptazy/ProxLB/discussions/)
 | GitHub | [ProxLB GitHub](https://github.com/gyptazy/ProxLB/issues) |
 
 ### Author(s)
- * Florian Paul Azim Hoberg @gyptazy (https://gyptazy.ch)
+ * Florian Paul Azim Hoberg @gyptazy (https://gyptazy.com)

docs/02_Configuration.md

@@ -23,4 +23,26 @@ Afterwards, restart the service (if running in daemon mode) to activate this reb
 <img align="left" src="https://cdn.gyptazy.ch/images/plb-rebalancing-exclude-balance-group.jpg"/> Access the Proxmox Web UI by opening your web browser and navigating to your Proxmox VE web interface, then log in with your credentials. Navigate to the VM you want to tag by selecting it from the left-hand navigation panel. Click on the "Options" tab to view the VM's options, then select "Edit" or "Add" (depending on whether you are editing an existing tag or adding a new one). In the tag field, enter plb_exclude_ followed by your unique identifier, for example, plb_exclude_critical. Save the changes to apply the tag to the VM. Repeat these steps for each VM that should be excluded from being on the same node.
 
 ### Ignore VMs (tag style)
-<img align="left" src="https://cdn.gyptazy.ch/images/plb-rebalancing-ignore-vm.jpg"/>  In Proxmox, you can ensure that certain VMs are ignored during the rebalancing process by setting a specific tag within the Proxmox Web UI, rather than solely relying on configurations in the ProxLB config file. This can be achieved by adding the tag 'plb_ignore_vm' to the VM. Once this tag is applied, the VM will be excluded from any further rebalancing operations, simplifying the management process.
+<img align="left" src="https://cdn.gyptazy.ch/images/plb-rebalancing-ignore-vm.jpg"/>  In Proxmox, you can ensure that certain VMs are ignored during the rebalancing process by setting a specific tag within the Proxmox Web UI, rather than solely relying on configurations in the ProxLB config file. This can be achieved by adding the tag 'plb_ignore_vm' to the VM. Once this tag is applied, the VM will be excluded from any further rebalancing operations, simplifying the management process.
+
+## Authentication / User Account / User / Permissions
+### Authentication
+ProxLB also supports different accounts in ProxLB. Therefore, you can simply create a new user and group and add the required roles permissions.
+
+### Creating Dedicated User for Balanciung
+It is recommended to not use the `root@pam` user for balancing. Therefore, creating a new user might be suitable and is very easy to create.
+A new user can be created by the gui, api and cli. The required roles are stated in the next chapter, but you can also use the following lines
+to create a user on the cli with the required roles to balance VMs & CTs.
+
+```
+pveum role add ProxLBAdmin --privs Datastore.Audit,Sys.Audit,VM.Audit,VM.Migrate
+pveum user add proxlb_admin@pve --password <password>
+pveum acl modify / --roles ProxLBAdmin --users proxlb_admin@pve
+```
+
+### Required Roles
+When using ProxLB with a dedicated account, you might also keep the assigned roles low. Therefore, you need to ensure that the newly created user is at least assigned to the following roles:
+* Datastore.Audit (Required for storage evaluation)
+* Sys.Audit (Required to get resource metrics of the nodes)
+* VM.Audit  (Requited to get resource metrics of VMs/CTs)
+* VM.Migrate (Required for migration of VMs/CTs)

packaging/CMakeLists.txt

@@ -1,5 +1,5 @@
 cmake_minimum_required(VERSION 3.16)
-project(proxmox-rebalancing-service VERSION 1.0.2)
+project(proxmox-rebalancing-service VERSION 1.0.4)
 
 install(PROGRAMS ../proxlb DESTINATION /bin)
 install(FILES ../proxlb.conf DESTINATION /etc/proxlb)

packaging/changelog_debian

@@ -1,3 +1,35 @@
+proxlb (1.0.4) unstable; urgency=low
+
+  * Add feature to make API timeout configureable.
+  * Add maintenance mode to evacuate a node and move workloads for other nodes in the cluster.
+  * Add version output cli arg.
+  * Run storage balancing only on supported shared storages.
+  * Run storage balancing only when needed to save time.
+  * Fix CPU balancing where calculations are done in float instead of int. (by @glitchvern)
+  * Fix documentation for the underlying infrastructure.
+
+ -- Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>  Fri, 11 Oct 2024 06:14:13 +0200
+
+proxlb (1.0.3) unstable; urgency=low
+
+  * Add a convert function to cast all bool alike options from configparser to bools.
+  * Add a config parser options for future features.
+  * Add a config versio schema that must be supported by ProxLB.
+  * Add feature to allow the API hosts being provided as a comma separated list.
+  * Add storage balancing function.
+  * Add doc how to add dedicated user for authentication. (by @Dulux-Oz)
+  * Add cli arg `-b` to return the next best node for next VM/CT placement.Fix some wonkey code styles.
+  * Provide a more reasonable output when HA services are not active in a Proxmox cluster.
+  * Improve the underlying code base for future implementations.
+  * Fix documentation for the master_only parameter placed in the wrong config section.
+  * Fixed `master_only` function by inverting the condition.
+  * Improved the overall validation and error handling.
+  * Fix bug in the `proxlb.conf` in the vm_balancing section.
+  * Fix handling of unset `ignore_nodes` and `ignore_vms` resulted in an attribute error.
+  * Fix anti-affinity rules not evaluating a new and different node.
+
+ -- Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>  Wed, 11 Sep 2024 17:31:03 +0200
+
 proxlb (1.0.2) unstable; urgency=low
 
   * Add option to run migration in parallel or sequentially.

packaging/changelog_redhat

@@ -1,3 +1,29 @@
+* Fri Oct 11 2024 Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>
+- Add feature to make API timeout configureable.
+- Add maintenance mode to evacuate a node and move workloads for other nodes in the cluster.
+- Add version output cli arg.
+- Run storage balancing only on supported shared storages.
+- Run storage balancing only when needed to save time.
+- Fix CPU balancing where calculations are done in float instead of int. (by @glitchvern)
+- Fix documentation for the underlying infrastructure.
+
+* Wed Sep 12 2024 Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>
+- Add a convert function to cast all bool alike options from configparser to bools.
+- Add a config parser options for future features.
+- Add a config versio schema that must be supported by ProxLB.
+- Add feature to allow the API hosts being provided as a comma separated list.
+- Add storage balancing function.
+- Add doc how to add dedicated user for authentication. (by @Dulux-Oz)
+- Add cli arg `-b` to return the next best node for next VM/CT placement.Fix some wonkey code styles.
+- Provide a more reasonable output when HA services are not active in a Proxmox cluster.
+- Improve the underlying code base for future implementations.
+- Fix documentation for the master_only parameter placed in the wrong config section.
+- Fixed `master_only` function by inverting the condition.
+- Improved the overall validation and error handling.
+- Fix bug in the `proxlb.conf` in the vm_balancing section.
+- Fix handling of unset `ignore_nodes` and `ignore_vms` resulted in an attribute error.
+- Fix anti-affinity rules not evaluating a new and different node.
+
 * Tue Aug 13 2024 Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>
 - Add option to run migration in parallel or sequentially.
 - Add option to run ProxLB only on a Proxmox cluster master (req. HA feature).

proxlb

@@ -42,7 +42,7 @@ import urllib3
 
 # Constants
 __appname__        = "ProxLB"
-__version__        = "1.0.3b"
+__version__        = "1.0.4"
 __config_version__ = 3
 __author__         = "Florian Paul Azim Hoberg <gyptazy@gyptazy.ch> @gyptazy"
 __errors__         = False
@@ -89,13 +89,19 @@ def initialize_logger(log_level, update_log_verbosity=False):
         logging.info(f'{info_prefix} Logger verbosity got updated to: {log_level}.')
 
 
-def pre_validations(config_path):
+def pre_validations(config_path, proxlb_config=False):
     """ Run pre-validations as sanity checks. """
     info_prefix = 'Info: [pre-validations]:'
 
-    __validate_imports()
-    __validate_config_file(config_path)
-    logging.info(f'{info_prefix} All pre-validations done.')
+    if proxlb_config:
+        logging.info(f'{info_prefix} Validating ProxLB config file content.')
+        __validate_config_content(proxlb_config)
+        logging.info(f'{info_prefix} ProxLB config file content validation done.')
+    else:
+        logging.info(f'{info_prefix} Validating basic configuration.')
+        __validate_imports()
+        __validate_config_file(config_path)
+        logging.info(f'{info_prefix} All pre-validations done.')
 
 
 def post_validations():
@@ -145,16 +151,75 @@ def __validate_config_file(config_path):
         logging.info(f'{info_prefix} Configuration file loaded from: {config_path}.')
 
 
+def __validate_config_content(proxlb_config):
+    """ Validate the user's config options. """
+    error_prefix = 'Error: [config]:'
+    info_prefix  = 'Info: [config]:'
+
+    validate_bool_options = [
+        'proxmox_api_ssl_v',
+        'vm_balancing_enable',
+        'vm_parallel_migrations',
+        'storage_balancing_enable',
+        'storage_parallel_migrations',
+        'update_service',
+        'api',
+        'master_only',
+        'daemon'
+    ]
+
+    for bool_val in validate_bool_options:
+        if type(proxlb_config.get(bool_val, None)) == bool:
+            logging.info(f'{info_prefix} Config option {bool_val} is in a correct format.')
+        else:
+            logging.critical(f'{error_prefix} Config option {bool_val} is incorrect: {proxlb_config.get(bool_val, None)}')
+            sys.exit(2)
+
+    validate_string_options = [
+        'vm_balancing_method',
+        'vm_balancing_mode',
+        'vm_balancing_mode_option',
+        'vm_balancing_type',
+        'storage_balancing_method',
+        'log_verbosity'
+    ]
+
+    whitelist_string_options = {
+        'vm_balancing_method': ['memory', 'disk', 'cpu'],
+        'vm_balancing_mode': ['used', 'assigned'],
+        'vm_balancing_mode_option': ['bytes', 'percent'],
+        'vm_balancing_type': ['vm', 'ct', 'all'],
+        'storage_balancing_method': ['disk_space'],
+        'log_verbosity': ['INFO', 'CRITICAL']
+    }
+
+    for string_val in validate_string_options:
+        if proxlb_config[string_val] in whitelist_string_options[string_val]:
+            logging.info(f'{info_prefix} Config option {string_val} is in a correct format.')
+        else:
+            logging.critical(f'{error_prefix} Config option {string_val} is incorrect: {proxlb_config.get(string_val, None)}')
+            sys.exit(2)
+
+
 def initialize_args():
     """ Initialize given arguments for ProxLB. """
     argparser = argparse.ArgumentParser(description='ProxLB')
-    argparser.add_argument('-c', '--config',    type=str, help='Path to config file.', required=False)
-    argparser.add_argument('-d', '--dry-run',   help='Perform a dry-run without doing any actions.', action='store_true', required=False)
-    argparser.add_argument('-j', '--json',      help='Return a JSON of the VM movement.',            action='store_true', required=False)
-    argparser.add_argument('-b', '--best-node', help='Returns the best next node.',                  action='store_true', required=False)
+    argparser.add_argument('-c', '--config',      help='Path to config file',                                   type=str,            required=False)
+    argparser.add_argument('-d', '--dry-run',     help='Perform a dry-run without doing any actions.',          action='store_true', required=False)
+    argparser.add_argument('-j', '--json',        help='Return a JSON of the VM movement.',                     action='store_true', required=False)
+    argparser.add_argument('-b', '--best-node',   help='Returns the best next node.',                           action='store_true', required=False)
+    argparser.add_argument('-m', '--maintenance', help='Sets node to maintenance mode & moves workloads away.', type=str,            required=False)
+    argparser.add_argument('-v', '--version',     help='Returns the current ProxLB version.',                   action='store_true', required=False)
     return argparser.parse_args()
 
 
+def proxlb_output_version():
+    """ Print ProxLB version information on CLI. """
+    print(f'{__appname__} version {__version__}\nRequired config version: >= {__config_version__}')
+    print('ProxLB support: https://github.com/gyptazy/ProxLB\nDeveloper: gyptazy.com')
+    sys.exit(0)
+
+
 def initialize_config_path(app_args):
     """ Initialize path to ProxLB config file. """
     info_prefix  = 'Info: [config]:'
@@ -182,6 +247,7 @@ def initialize_config_options(config_path):
         proxlb_config['proxmox_api_user']            = config['proxmox']['api_user']
         proxlb_config['proxmox_api_pass']            = config['proxmox']['api_pass']
         proxlb_config['proxmox_api_ssl_v']           = config['proxmox']['verify_ssl']
+        proxlb_config['proxmox_api_timeout']         = config['proxmox'].get('timeout', 10)
         # VM Balancing
         proxlb_config['vm_balancing_enable']         = config['vm_balancing'].get('enable', 1)
         proxlb_config['vm_balancing_method']         = config['vm_balancing'].get('method', 'memory')
@@ -190,8 +256,10 @@ def initialize_config_options(config_path):
         proxlb_config['vm_balancing_type']           = config['vm_balancing'].get('type', 'vm')
         proxlb_config['vm_balanciness']              = config['vm_balancing'].get('balanciness', 10)
         proxlb_config['vm_parallel_migrations']      = config['vm_balancing'].get('parallel_migrations', 1)
-        proxlb_config['vm_ignore_nodes']             = config['vm_balancing'].get('ignore_nodes', None)
-        proxlb_config['vm_ignore_vms']               = config['vm_balancing'].get('ignore_vms', None)
+        proxlb_config['vm_maintenance_nodes']        = config['vm_balancing'].get('maintenance_nodes', '')
+        proxlb_config['vm_ignore_nodes']             = config['vm_balancing'].get('ignore_nodes', '')
+        proxlb_config['vm_ignore_vms']               = config['vm_balancing'].get('ignore_vms', '')
+        proxlb_config['vm_enforce_affinity_groups']  = config['vm_balancing'].get('enforce_affinity_groups', 1)
         # Storage Balancing
         proxlb_config['storage_balancing_enable']    = config['storage_balancing'].get('enable', 0)
         proxlb_config['storage_balancing_method']    = config['storage_balancing'].get('method', 'disk_space')
@@ -256,7 +324,7 @@ def validate_config_minimum_version(proxlb_config):
         logging.info(f'{info_prefix} ProxLB config version {proxlb_config["config_version"]} is fine. Required: {__config_version__}.')
 
 
-def api_connect(proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v):
+def api_connect(proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v, proxmox_api_timeout):
     """ Connect and authenticate to the Proxmox remote API. """
     error_prefix = 'Error: [api-connection]:'
     warn_prefix  = 'Warning: [api-connection]:'
@@ -270,7 +338,10 @@ def api_connect(proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_ap
     proxmox_api_host = __api_connect_get_host(proxmox_api_host)
 
     try:
-        api_object = proxmoxer.ProxmoxAPI(proxmox_api_host, user=proxmox_api_user, password=proxmox_api_pass, verify_ssl=proxmox_api_ssl_v)
+        api_object = proxmoxer.ProxmoxAPI(proxmox_api_host, user=proxmox_api_user, password=proxmox_api_pass, verify_ssl=proxmox_api_ssl_v, timeout=int(proxmox_api_timeout))
+    except proxmoxer.backends.https.AuthenticationError as proxmox_api_error:
+        logging.critical(f'{error_prefix} Provided credentials do not work: {proxmox_api_error}')
+        sys.exit(2)
     except urllib3.exceptions.NameResolutionError:
         logging.critical(f'{error_prefix} Could not resolve the given host: {proxmox_api_host}.')
         sys.exit(2)
@@ -306,6 +377,7 @@ def __api_connect_get_host(proxmox_api_host):
 
 
 def __api_connect_test_ipv4_host(proxmox_api_host, port):
+    """ Validate if a given host on the IPv4 management address is reachable. """
     error_prefix               = 'Error: [api-connect-test-host]:'
     info_prefix                = 'Info: [api-connect-test-host]:'
     proxmox_connection_timeout = 2
@@ -326,6 +398,7 @@ def __api_connect_test_ipv4_host(proxmox_api_host, port):
 
 
 def __api_connect_test_ipv6_host(proxmox_api_host, port):
+    """ Validate if a given host on the IPv6 management address is reachable. """
     error_prefix               = 'Error: [api-connect-test-host]:'
     info_prefix                = 'Info: [api-connect-test-host]:'
     proxmox_connection_timeout = 2
@@ -383,7 +456,7 @@ def get_cluster_master(api_object):
     if cluster_master:
         return cluster_master
     else:
-        logging.critical(f'{error_prefix} Could not obtain cluster master. Please check your configuration - stopping.')
+        logging.critical(f'{error_prefix} Could not obtain cluster master. Please check your configuration and ensure HA services in Proxmox are enabled. Stopping.')
         sys.exit(2)
 
 
@@ -401,21 +474,24 @@ def validate_cluster_master(cluster_master):
         return True
 
 
-def get_node_statistics(api_object, ignore_nodes):
+def get_node_statistics(api_object, ignore_nodes, maintenance_nodes):
     """ Get statistics of cpu, memory and disk for each node in the cluster. """
-    info_prefix       = 'Info: [node-statistics]:'
-    node_statistics   = {}
-    ignore_nodes_list =  ignore_nodes.split(',')
+    info_prefix            = 'Info: [node-statistics]:'
+    node_statistics        = {}
+    ignore_nodes_list      =  ignore_nodes.split(',')
+    maintenance_nodes_list =  maintenance_nodes.split(',')
 
     for node in api_object.nodes.get():
-        if node['status'] == 'online' and node['node'] not in ignore_nodes_list:
+        if node['status'] == 'online':
             node_statistics[node['node']] = {}
+            node_statistics[node['node']]['maintenance']                      = False
+            node_statistics[node['node']]['ignore']                           = False
             node_statistics[node['node']]['cpu_total']                        = node['maxcpu']
-            node_statistics[node['node']]['cpu_assigned']                     = node['cpu']
+            node_statistics[node['node']]['cpu_assigned']                     = 0
             node_statistics[node['node']]['cpu_assigned_percent']             = int((node_statistics[node['node']]['cpu_assigned']) / int(node_statistics[node['node']]['cpu_total']) * 100)
             node_statistics[node['node']]['cpu_assigned_percent_last_run']    = 0
-            node_statistics[node['node']]['cpu_used']                         = 0
-            node_statistics[node['node']]['cpu_free']                         = int(node['maxcpu']) - int(node['cpu'])
+            node_statistics[node['node']]['cpu_used']                         = node['cpu']
+            node_statistics[node['node']]['cpu_free']                         = (node['maxcpu']) - (node['cpu'] * node['maxcpu'])
             node_statistics[node['node']]['cpu_free_percent']                 = int((node_statistics[node['node']]['cpu_free']) / int(node['maxcpu']) * 100)
             node_statistics[node['node']]['cpu_free_percent_last_run']        = 0
             node_statistics[node['node']]['memory_total']                     = node['maxmem']
@@ -436,6 +512,15 @@ def get_node_statistics(api_object, ignore_nodes):
             node_statistics[node['node']]['disk_free_percent_last_run']       = 0
             logging.info(f'{info_prefix} Added node {node["node"]}.')
 
+            # Update node specific vars
+            if node['node'] in maintenance_nodes_list:
+                node_statistics[node['node']]['maintenance']                      = True
+                logging.info(f'{info_prefix} Maintenance mode: {node["node"]} is set to maintenance mode.')
+
+            if node['node'] in ignore_nodes_list:
+                node_statistics[node['node']]['ignore']                           = True
+                logging.info(f'{info_prefix} Ignore Node: {node["node"]} is set to be ignored.')
+
     logging.info(f'{info_prefix} Created node statistics.')
     return node_statistics
 
@@ -445,7 +530,7 @@ def get_vm_statistics(api_object, ignore_vms, balancing_type):
     info_prefix                 = 'Info: [vm-statistics]:'
     warn_prefix                 = 'Warn: [vm-statistics]:'
     vm_statistics               = {}
-    ignore_vms_list             =  ignore_vms.split(',')
+    ignore_vms_list             = ignore_vms.split(',')
     group_include               = None
     group_exclude               = None
     vm_ignore                   = None
@@ -620,6 +705,7 @@ def update_node_statistics(node_statistics, vm_statistics):
 def get_storage_statistics(api_object):
     """ Get statistics of all storage in the cluster. """
     info_prefix        = 'Info: [storage-statistics]:'
+    storage_whitelist  = ['nfs']
     storage_statistics = {}
 
     for node in api_object.nodes.get():
@@ -628,7 +714,7 @@ def get_storage_statistics(api_object):
 
             # Only add enabled and active storage repositories that might be suitable for further
             # storage balancing.
-            if storage['enabled'] and storage['active'] and storage['shared']:
+            if storage['enabled'] and storage['active'] and storage['shared'] and storage['type'] in storage_whitelist:
                 storage_statistics[storage['storage']]                          = {}
                 storage_statistics[storage['storage']]['name']                  = storage['storage']
                 storage_statistics[storage['storage']]['total']                 = storage['total']
@@ -692,7 +778,10 @@ def __get_vm_tags(api_object, node, vmid, balancing_type):
     if balancing_type == 'ct':
         vm_config = api_object.nodes(node['node']).lxc(vmid).config.get()
 
-    logging.info(f'{info_prefix} Got VM/CT tag from API.')
+    if vm_config.get("tags", None) is None:
+        logging.info(f'{info_prefix} Got no VM/CT tag for VM {vm_config.get("name", None)} from API.')
+    else:
+        logging.info(f'{info_prefix} Got VM/CT tag {vm_config.get("tags", None)} for VM  {vm_config.get("name", None)} from API.')
     return vm_config.get('tags', None)
 
 
@@ -710,8 +799,16 @@ def __get_proxlb_groups(vm_tags):
             logging.info(f'{info_prefix} Got PLB include group.')
             group_include = group
 
-        if group.startswith('plb_exclude_'):
+        if group.startswith('plb_affinity_'):
             logging.info(f'{info_prefix} Got PLB include group.')
+            group_include = group
+
+        if group.startswith('plb_exclude_'):
+            logging.info(f'{info_prefix} Got PLB exclude group.')
+            group_exclude = group
+
+        if group.startswith('plb_antiaffinity_'):
+            logging.info(f'{info_prefix} Got PLB exclude group.')
             group_exclude = group
 
         if group.startswith('plb_ignore_vm'):
@@ -731,11 +828,17 @@ def balancing_vm_calculations(balancing_method, balancing_mode, balancing_mode_o
     __validate_vm_statistics(vm_statistics)
     rebalance = __validate_balanciness(balanciness, balancing_method, balancing_mode, node_statistics)
 
+    # Run rebalancing calculations.
     if rebalance:
         # Get most used/assigned resources of the VM and the most free or less allocated node.
         resources_vm_most_used, processed_vms  = __get_most_used_resources_vm(balancing_method, balancing_mode, vm_statistics, processed_vms)
         resources_node_most_free               = __get_most_free_resources_node(balancing_method, balancing_mode, balancing_mode_option, node_statistics)
 
+        # If most used vm is on most free node then skip it and get another one.
+        while resources_vm_most_used[1]['node_parent'] == resources_node_most_free[0] and len(processed_vms) < len(vm_statistics):
+            resources_vm_most_used, processed_vms  = __get_most_used_resources_vm(balancing_method, balancing_mode, vm_statistics, processed_vms)
+            logging.debug(f'{info_prefix} processed {len(processed_vms)} out of {len(vm_statistics)} vms.')
+
         # Update resource statistics for VMs and nodes.
         node_statistics, vm_statistics         = __update_vm_resource_statistics(resources_vm_most_used, resources_node_most_free,
                                                                              vm_statistics, node_statistics, balancing_method, balancing_mode)
@@ -752,14 +855,42 @@ def balancing_vm_calculations(balancing_method, balancing_mode, balancing_mode_o
         logging.info(f'{info_prefix} Best next node for VM & CT placement: {best_next_node[0]}')
         sys.exit(0)
 
-    # Honour groupings for include and exclude groups for rebalancing VMs.
-    node_statistics, vm_statistics = __get_vm_tags_include_groups(vm_statistics, node_statistics, balancing_method, balancing_mode)
-    node_statistics, vm_statistics = __get_vm_tags_exclude_groups(vm_statistics, node_statistics, balancing_method, balancing_mode)
-
     logging.info(f'{info_prefix} Balancing calculations done.')
     return node_statistics, vm_statistics
 
 
+def balancing_vm_maintenance(proxlb_config, app_args, node_statistics, vm_statistics):
+    """ Calculate re-balancing of VMs that need to be moved away from maintenance nodes. """
+    info_prefix            = 'Info: [rebalancing-maintenance-vm-calculator]:'
+    maintenance_nodes_list = proxlb_config['vm_maintenance_nodes'].split(',')
+    nodes_present          = list(node_statistics.keys())
+    balancing_method       = proxlb_config['vm_balancing_method']
+    balancing_mode         = proxlb_config['vm_balancing_mode']
+    balancing_mode_option  = proxlb_config['vm_balancing_mode_option']
+
+    # Merge maintenance nodes from config and cli args.
+    if app_args.maintenance is not None:
+        logging.info(f'{info_prefix} Maintenance nodes from CLI arg and config will be merged.')
+        maintenance_nodes_list = maintenance_nodes_list + app_args.maintenance.split(',')
+
+    # Ensure that only existing nodes in the cluster will be used.
+    if len(proxlb_config['vm_maintenance_nodes']) > 1:
+        maintenance_nodes_list = set(maintenance_nodes_list) & set(nodes_present)
+        logging.info(f'{info_prefix} Maintenance mode for the following hosts defined: {maintenance_nodes_list}')
+    else:
+        logging.info(f'{info_prefix} No nodes for maintenance mode defined.')
+        return node_statistics, vm_statistics
+
+    for node_name in maintenance_nodes_list:
+        node_vms = sorted(vm_statistics.items(), key=lambda item: item[0] if item[1]['node_parent'] == node_name else [])
+        # Update resource statistics for VMs and nodes.
+        for vm in node_vms:
+            resources_node_most_free        = __get_most_free_resources_node(balancing_method, balancing_mode, balancing_mode_option, node_statistics)
+            node_statistics, vm_statistics  = __update_vm_resource_statistics(vm, resources_node_most_free, vm_statistics, node_statistics, balancing_method, balancing_mode)
+
+    return node_statistics, vm_statistics
+
+
 def __validate_balancing_method(balancing_method):
     """ Validate for valid and supported balancing method. """
     error_prefix = 'Error: [balancing-method-validation]:'
@@ -863,11 +994,11 @@ def __get_most_free_resources_node(balancing_method, balancing_mode, balancing_m
 
     # Return the node information based on the balancing mode.
     if balancing_mode == 'used' and balancing_mode_option == 'bytes':
-        node = max(node_statistics.items(), key=lambda item: item[1][f'{balancing_method}_free'])
+        node = max(node_statistics.items(), key=lambda item: item[1][f'{balancing_method}_free'] if not item[1]['maintenance'] else -float('inf'))
     if balancing_mode == 'used' and balancing_mode_option == 'percent':
-        node = max(node_statistics.items(), key=lambda item: item[1][f'{balancing_method}_free_percent'])
+        node = max(node_statistics.items(), key=lambda item: item[1][f'{balancing_method}_free_percent'] if not item[1]['maintenance'] else -float('inf'))
     if balancing_mode == 'assigned':
-        node = min(node_statistics.items(), key=lambda item: item[1][f'{balancing_method}_assigned'] if item[1][f'{balancing_method}_assigned_percent'] > 0 or item[1][f'{balancing_method}_assigned_percent'] < 100 else -float('inf'))
+        node = min(node_statistics.items(), key=lambda item: item[1][f'{balancing_method}_assigned'] if not item[1]['maintenance'] and (item[1][f'{balancing_method}_assigned_percent'] > 0 or item[1][f'{balancing_method}_assigned_percent'] < 100) else -float('inf'))
 
     logging.info(f'{info_prefix} {node}')
     return node
@@ -888,7 +1019,7 @@ def __update_vm_resource_statistics(resource_highest_used_resources_vm, resource
         # Assign new rebalance node to vm
         vm_statistics[vm_name]['node_rebalance'] = vm_node_rebalance
 
-        logging.info(f'Moving {vm_name} from {vm_node_parent} to {vm_node_rebalance}')
+        logging.info(f'{info_prefix} Moving {vm_name} from {vm_node_parent} to {vm_node_rebalance}')
 
         # Recalculate values for nodes
         ## Add freed resources to old parent node
@@ -946,43 +1077,73 @@ def __get_vm_tags_include_groups(vm_statistics, node_statistics, balancing_metho
 
 def __get_vm_tags_exclude_groups(vm_statistics, node_statistics, balancing_method, balancing_mode):
     """ Get VMs tags for exclude groups. """
-    info_prefix = 'Info: [rebalancing-tags-group-exclude]:'
+    info_prefix      = 'Info: [rebalancing-tags-group-exclude]:'
     tags_exclude_vms = {}
-    processed_vm = []
 
     # Create groups of tags with belongings hosts.
     for vm_name, vm_values in vm_statistics.items():
-        if vm_values.get('group_include', None):
-            if not tags_exclude_vms.get(vm_values['group_include'], None):
-                tags_exclude_vms[vm_values['group_include']] = [vm_name]
+        if vm_values.get('group_exclude', None):
+            if not tags_exclude_vms.get(vm_values['group_exclude'], None):
+                tags_exclude_vms[vm_values['group_exclude']]               = {}
+                tags_exclude_vms[vm_values['group_exclude']]['nodes_used'] = []
+                tags_exclude_vms[vm_values['group_exclude']]['nodes_used'].append(vm_statistics[vm_name]['node_rebalance'])
+                tags_exclude_vms[vm_values['group_exclude']]['vms']        = [vm_name]
             else:
-                tags_exclude_vms[vm_values['group_include']] = tags_exclude_vms[vm_values['group_include']] + [vm_name]
+                tags_exclude_vms[vm_values['group_exclude']]['vms']        = tags_exclude_vms[vm_values['group_exclude']]['vms'] + [vm_name]
+                tags_exclude_vms[vm_values['group_exclude']]['nodes_used'].append(vm_statistics[vm_name]['node_rebalance'])
 
-    # Update the VMs to the corresponding node to their group assignments.
-    for group, vm_names in tags_exclude_vms.items():
-        # Do not take care of tags that have only a single host included.
-        if len(vm_names) < 2:
-            logging.info(f'{info_prefix} Only one host in group assignment.')
-            return node_statistics, vm_statistics
+    # Evaluate all VMs assigned for each exclude groups and validate that they will be moved to another random node.
+    # However, if there are still more VMs than nodes we need to deal with it.
+    for exclude_group, group_values in tags_exclude_vms.items():
+
+        group_values['nodes_used'] = []
+        for vm in group_values['vms']:
+
+            proceed = True
+            counter = 0
+
+            while proceed:
+
+                if vm_statistics[vm]['node_rebalance'] in group_values['nodes_used']:
+                    # Find another possible new target node if possible by randomly get any node from
+                    # the cluster and validating if this is already used for this anti-affinity group.
+                    logging.info(f'{info_prefix} Rebalancing of VM {vm} is needed due to anti-affinity group policy.')
+                    random_node, counter, proceed = __get_random_node(counter, node_statistics, vm)
+
+                    if random_node not in group_values['nodes_used']:
+                        logging.info(f'{info_prefix} New random node {random_node} has not yet been used for the anti-affinity group {exclude_group}.')
+                        group_values['nodes_used'].append(random_node)
+                        logging.info(f'{info_prefix} New random node {random_node} has been added as an already used node to the anti-affinity group {exclude_group}.')
+                        logging.info(f'{info_prefix} VM {vm} switched node from {vm_statistics[vm]["node_rebalance"]} to {random_node} due to the anti-affinity group {exclude_group}.')
+                        vm_statistics[vm]['node_rebalance'] = random_node
 
-        vm_node_rebalance = False
-        logging.info(f'{info_prefix} Create exclude groups of VM hosts.')
-        for vm_name in vm_names:
-            if vm_name not in processed_vm:
-                if not vm_node_rebalance:
-                    random_node = vm_statistics[vm_name]['node_parent']
-                    # Get a random node and make sure that it is not by accident the
-                    # currently assigned one.
-                    while random_node == vm_statistics[vm_name]['node_parent']:
-                        random_node = random.choice(list(node_statistics.keys()))
                 else:
-                    _mocked_vm_object = (vm_name, vm_statistics[vm_name])
-                    node_statistics, vm_statistics = __update_vm_resource_statistics(_mocked_vm_object, [random_node], vm_statistics, node_statistics, balancing_method, balancing_mode)
-            processed_vm.append(vm_name)
+                    # Add the used node to the list for the anti-affinity group to ensure no
+                    # other VM with the same anti-affinity group will use it (if possible).
+                    logging.info(f'{info_prefix} Node {vm_statistics[vm]["node_rebalance"]} has been added as an already used node to the anti-affinity group {exclude_group}.')
+                    logging.info(f'{info_prefix} No rebalancing for VM {vm} needed due to any anti-affinity group policies.')
+                    group_values['nodes_used'].append(vm_statistics[vm]['node_rebalance'])
+                    proceed = False
 
     return node_statistics, vm_statistics
 
 
+def __get_random_node(counter, node_statistics, vm):
+    """ Get a random node within the Proxmox cluster. """
+    warning_prefix = 'Warning: [random-node-getter]:'
+    info_prefix    = 'Info: [random-node-getter]:'
+
+    counter = counter + 1
+    random_node = None
+    if counter < 30:
+        random_node = random.choice(list(node_statistics.keys()))
+        logging.info(f'{info_prefix} New random node {random_node} evaluated for vm {vm} in run {counter}.')
+        return random_node, counter, False
+    else:
+        logging.warning(f'{warning_prefix} Reached limit for random node evaluation for vm {vm}. Unable to find a suitable new node.')
+        return random_node, counter, False
+
+
 def __wait_job_finalized(api_object, node_name, job_id, counter):
     """ Wait for a job to be finalized. """
     error_prefix = 'Error: [job-status-getter]:'
@@ -1042,7 +1203,10 @@ def __run_vm_rebalancing(api_object, _vm_vm_statistics, app_args, parallel_migra
                 logging.info(f'{info_prefix} Rebalancing will be performed parallely.')
 
     else:
-        logging.info(f'{info_prefix} No rebalancing needed.')
+        if app_args.dry_run:
+            logging.info(f'{info_prefix} Running in dry run mode. Not executing any balancing.')
+        else:
+            logging.info(f'{info_prefix} No rebalancing needed.')
 
     return _vm_vm_statistics
 
@@ -1330,6 +1494,13 @@ def size_in_bytes(size_str):
     return size_value * size_multipliers.get(size_unit, 1)
 
 
+def balancing_vm_affinity_groups(node_statistics, vm_statistics, balancing_method, balancing_mode):
+    """ Enforce (anti-)affinity groups for further VM movement across the cluster. """
+    node_statistics, vm_statistics = __get_vm_tags_include_groups(vm_statistics, node_statistics, balancing_method, balancing_mode)
+    node_statistics, vm_statistics = __get_vm_tags_exclude_groups(vm_statistics, node_statistics, balancing_method, balancing_mode)
+    return node_statistics, vm_statistics
+
+
 def main():
     """ Run ProxLB for balancing VM workloads across a Proxmox cluster. """
     vm_output_statistics      = {}
@@ -1338,18 +1509,21 @@ def main():
     # Initialize PAS.
     initialize_logger('CRITICAL')
     app_args = initialize_args()
+    if app_args.version:
+        proxlb_output_version()
     config_path = initialize_config_path(app_args)
     pre_validations(config_path)
 
     # Parse global config.
     proxlb_config = initialize_config_options(config_path)
+    pre_validations(config_path, proxlb_config)
 
     # Overwrite logging handler with user defined log verbosity.
     initialize_logger(proxlb_config['log_verbosity'], update_log_verbosity=True)
 
     while True:
         # API Authentication.
-        api_object = api_connect(proxlb_config['proxmox_api_host'], proxlb_config['proxmox_api_user'], proxlb_config['proxmox_api_pass'], proxlb_config['proxmox_api_ssl_v'])
+        api_object = api_connect(proxlb_config['proxmox_api_host'], proxlb_config['proxmox_api_user'], proxlb_config['proxmox_api_pass'], proxlb_config['proxmox_api_ssl_v'], proxlb_config['proxmox_api_timeout'])
 
         # Get master node of cluster and ensure that ProxLB is only performed on the
         # cluster master node to avoid ongoing rebalancing.
@@ -1360,16 +1534,21 @@ def main():
             validate_daemon(proxlb_config['daemon'], proxlb_config['schedule'])
             continue
 
-        # Get metric & statistics for vms and nodes.
+        # Get metrics & statistics for vms and nodes.
         if proxlb_config['vm_balancing_enable'] or proxlb_config['storage_balancing_enable'] or app_args.best_node:
-            node_statistics    = get_node_statistics(api_object, proxlb_config['vm_ignore_nodes'])
+            node_statistics    = get_node_statistics(api_object, proxlb_config['vm_ignore_nodes'], proxlb_config['vm_maintenance_nodes'])
             vm_statistics      = get_vm_statistics(api_object, proxlb_config['vm_ignore_vms'], proxlb_config['vm_balancing_type'])
             node_statistics    = update_node_statistics(node_statistics, vm_statistics)
-            storage_statistics = get_storage_statistics(api_object)
+            # Obtaining metrics for the storage may take longer times and is not needed for VM/CT balancing.
+            # We can save time by skipping this when not really needed.
+            if proxlb_config['storage_balancing_enable']:
+                storage_statistics = get_storage_statistics(api_object)
 
         # Execute VM/CT balancing sub-routines.
         if proxlb_config['vm_balancing_enable'] or app_args.best_node:
             node_statistics, vm_statistics = balancing_vm_calculations(proxlb_config['vm_balancing_method'], proxlb_config['vm_balancing_mode'], proxlb_config['vm_balancing_mode_option'], node_statistics, vm_statistics, proxlb_config['vm_balanciness'], app_args, rebalance=False, processed_vms=[])
+            node_statistics, vm_statistics = balancing_vm_maintenance(proxlb_config, app_args, node_statistics, vm_statistics)
+            node_statistics, vm_statistics = balancing_vm_affinity_groups(node_statistics, vm_statistics, proxlb_config['vm_balancing_method'], proxlb_config['vm_balancing_mode'],)
             vm_output_statistics = run_rebalancing(api_object, vm_statistics, app_args, proxlb_config['vm_parallel_migrations'], 'vm')
 
         # Execute storage balancing sub-routines.

proxlb.conf

@@ -3,10 +3,11 @@ api_host: hypervisor01.gyptazy.ch
 api_user: root@pam
 api_pass: FooBar
 verify_ssl: 1
-[balancing]
+[vm_balancing]
 enable: 1
 method: memory
 mode: used
+maintenance_nodes: dummynode03,dummynode04
 ignore_nodes: dummynode01,dummynode02
 ignore_vms: testvm01,testvm02
 [storage_balancing]
@@ -18,4 +19,5 @@ enable: 0
 [service]
 daemon: 1
 schedule: 24
-log_verbosity: CRITICAL
+log_verbosity: CRITICAL
+config_version: 3