Merge pull request #50 from gyptazy/release/v1.0.2

release: Prepare release v1.0.2

.changelogs/1.0.0/17-add-configurable-log-verbosity.yml

@@ -1,4 +1,4 @@
 added:
    - Add feature to make log verbosity configurable [#17].
- changed:
+changed:
    - Adjusted general logging and log more details.

.changelogs/1.0.0/29_add_option_rebalance_by_node_percent.yml

@@ -0,0 +1,2 @@
+added:
+  - Add option_mode to rebalance by node's free resources in percent (instead of bytes). [#29]

.changelogs/1.0.0/release_meta.yml

@@ -1 +1 @@
-date: TBD
+date: 2024-08-01

.changelogs/1.0.2/40_add_option_to_run_only_on_cluster_master_node.yml

@@ -0,0 +1,2 @@
+added:
+  - Add option to run ProxLB only on the Proxmox's master node in the cluster (reg. HA feature). [#40]

.changelogs/1.0.2/41_add_option_run_migration_parallel_or_sequential.yml

@@ -0,0 +1,2 @@
+added:
+  - Add option to run migrations in parallel or sequentially. [#41]

.changelogs/1.0.2/45_fix_daemon_timer.yml

@@ -0,0 +1,2 @@
+changed:
+  - Fix daemon timer to use hours instead of minutes. [#45]

.changelogs/1.0.2/49_fix_cmake_debian_packaging.yml

@@ -0,0 +1,2 @@
+fixed:
+  - Fix CMake packaging for Debian package to avoid overwriting the config file. [#49]

.changelogs/1.0.2/release_meta.yml

@@ -0,0 +1 @@
+date: 2024-08-13

CHANGELOG.md

@@ -6,6 +6,40 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
+## [1.0.2] - 2024-08-13
+
+### Added
+
+- Add option to run migration in parallel or sequentially. [#41]
+- Add option to run ProxLB only on the Proxmox's master node in the cluster (reg. HA feature). [#40]
+
+### Changed
+
+- Fix daemon timer to use hours instead of minutes. [#45]
+- Fix CMake packaging for Debian package to avoid overwriting the config file. [#49]
+- Fix wonkey code style.
+
+
+## [1.0.0] - 2024-08-01
+
+### Added
+
+- Add feature to prevent VMs from being relocated by defining a wildcard pattern. [#7]
+- Add feature to make log verbosity configurable [#17].
+- Add option_mode to rebalance by node's free resources in percent (instead of bytes). [#29]
+- Add option to rebalance by assigned VM resources to avoid over provisioning. [#16]
+- Add Docker/Podman support. [#10 by @daanbosch]
+- Add exclude grouping feature to rebalance VMs from being located together to new nodes. [#4]
+- Add feature to prevent VMs from being relocated by defining the 'plb_ignore_vm' tag. [#7]
+- Add dry-run support to see what kind of rebalancing would be done. [#6]
+- Add LXC/Container integration. [#27]
+- Add include grouping feature to rebalance VMs bundled to new nodes. [#3]
+
+### Changed
+
+- Adjusted general logging and log more details.
+
+
 ## [0.9.9] - 2024-07-06
 
 ### Added

README.md

@@ -32,7 +32,10 @@
     - [Logging](#logging)
   - [Motivation](#motivation)
   - [References](#references)
-  - [Packages / Container Images](#packages--container-images)
+  - [Downloads](#downloads)
+    - [Packages](#packages)
+    - [Repository](#repository)
+    - [Container Images (Docker/Podman)](#container-images-dockerpodman)
   - [Misc](#misc)
     - [Bugs](#bugs)
     - [Contributing](#contributing)
@@ -102,11 +105,14 @@ The following options can be set in the `proxlb.conf` file:
 | api_pass | FooBar | Password for the API. |
 | verify_ssl | 1 | Validate SSL certificates (1) or ignore (0). (default: 1) |
 | method | memory | Defines the balancing method (default: memory) where you can use `memory`, `disk` or `cpu`. |
-| mode | used | Rebalance by `used` resources (efficiency) or `assigned` (avoid overprovisioning) resources. (default: used)|
-| type | vm | Rebalance only `vm` (virtual machines), `ct` (containers) or `all` (virtual machines & containers). (default: vm)|
+| mode | used | Rebalance by `used` resources (efficiency) or `assigned` (avoid overprovisioning) resources. (default: used)|
+| mode_option | byte | Rebalance by node's resources in `bytes` or `percent`. (default: bytes) |
+| type | vm | Rebalance only `vm` (virtual machines), `ct` (containers) or `all` (virtual machines & containers). (default: vm)|
 | balanciness | 10 | Value of the percentage of lowest and highest resource consumption on nodes may differ before rebalancing. (default: 10) |
+| parallel_migrations | 1 | Defines if migrations should be done parallely or sequentially. (default: 1) |
 | ignore_nodes | dummynode01,dummynode02,test* | Defines a comma separated list of nodes to exclude. |
 | ignore_vms | testvm01,testvm02 | Defines a comma separated list of VMs to exclude. (`*` as suffix wildcard or tags are also supported) |
+| master_only | 0 | Defines is this should only be performed (1) on the cluster master node or not (0). (default: 0) |
 | daemon | 1 | Run as a daemon (1) or one-shot (0). (default: 1) |
 | schedule | 24 | Hours to rebalance in hours. (default: 24) |
 | log_verbosity | INFO | Defines the log level (default: CRITICAL) where you can use `INFO`, `WARN` or `CRITICAL` |
@@ -129,9 +135,16 @@ type: vm
 # Rebalancing:     node01: 41% memory consumption :: node02: 52% consumption
 # No rebalancing:  node01: 43% memory consumption :: node02: 50% consumption
 balanciness: 10
+# Enable parallel migrations. If set to 0 it will wait for completed migrations
+# before starting next migration.
+parallel_migrations: 1
 ignore_nodes: dummynode01,dummynode02
 ignore_vms: testvm01,testvm02
 [service]
+# The master_only option might be usuful if running ProxLB on all nodes in a cluster
+# but only a single one should do the balancing. The master node is obtained from the Proxmox
+# HA status.
+master_only: 0
 daemon: 1
 ```
 
@@ -197,8 +210,8 @@ The executable must be able to read the config file, if no dedicated config file
 The easiest way to get started is by using the ready-to-use packages that I provide on my CDN and to run it on a Linux Debian based system. This can also be one of the Proxmox nodes itself.
 
 ```
-wget https://cdn.gyptazy.ch/files/amd64/debian/proxlb/proxlb_0.9.9_amd64.deb
-dpkg -i proxlb_0.9.9_amd64.deb
+wget https://cdn.gyptazy.ch/files/amd64/debian/proxlb/proxlb_1.0.2_amd64.deb
+dpkg -i proxlb_1.0.2_amd64.deb
 # Adjust your config
 vi /etc/proxlb/proxlb.conf
 systemctl restart proxlb
@@ -211,7 +224,7 @@ Creating a container image of ProxLB is straightforward using the provided Docke
 ```bash
 git clone https://github.com/gyptazy/ProxLB.git
 cd ProxLB
-build -t proxlb .
+docker build -t proxlb .
 ```
 
 Afterwards simply adjust the config file to your needs:
@@ -254,18 +267,45 @@ Here you can find some overviews of references for and about the ProxLB (PLB):
 | General introduction into ProxLB | https://gyptazy.ch/blog/proxlb-rebalancing-vm-workloads-across-nodes-in-proxmox-clusters/ |
 | Howto install and use ProxLB on Debian to rebalance vm workloads in a Proxmox cluster | https://gyptazy.ch/howtos/howto-install-and-use-proxlb-to-rebalance-vm-workloads-across-nodes-in-proxmox-clusters/ |
 
-## Packages / Container Images
+## Downloads
+ProxLB can be obtained in man different ways, depending on which use case you prefer. You can use simply copy the code from GitHub, use created packages for Debian or RedHat based systems, use a Repository to keep ProxLB always up to date or simply use a Container image for Docker/Podman.
+
+### Packages
 Ready to use packages can be found at:
 * https://cdn.gyptazy.ch/files/amd64/debian/proxlb/
 * https://cdn.gyptazy.ch/files/amd64/ubuntu/proxlb/
 * https://cdn.gyptazy.ch/files/amd64/redhat/proxlb/
 * https://cdn.gyptazy.ch/files/amd64/freebsd/proxlb/
 
+
+### Repository
+Debian based systems can also use the repository by adding the following line to their apt sources:
+
+```
+deb https://repo.gyptazy.ch/ /
+```
+
+The Repository's GPG key can be found at: `https://repo.gyptazy.ch/repo/KEY.gpg`
+
+You can also simply import it by running:
+
+```
+# KeyID:  DEB76ADF7A0BAADB51792782FD6A7A70C11226AA
+# SHA256: 5e44fffa09c747886ee37cc6e9e7eaf37c6734443cc648eaf0a9241a89084383  KEY.gpg
+
+wget -O /etc/apt/trusted.gpg.d/proxlb.asc https://repo.gyptazy.ch/repo/KEY.gpg
+```
+
+*Note: The defined repositories `repo.gyptazy.ch` and `repo.proxlb.de` are the same!*
+
+### Container Images (Docker/Podman)
 Container Images for Podman, Docker etc., can be found at:
 | Version | Image |
 |------|:------:|
 | latest | cr.gyptazy.ch/proxlb/proxlb:latest |
-| v0.0.9 | cr.gyptazy.ch/proxlb/proxlb:v0.0.9 |
+| v1.0.2 | cr.gyptazy.ch/proxlb/proxlb:v1.0.2 |
+| v1.0.0 | cr.gyptazy.ch/proxlb/proxlb:v1.0.0 |
+| v0.9.9 | cr.gyptazy.ch/proxlb/proxlb:v0.9.9 |
 
 ## Misc
 ### Bugs

packaging/CMakeLists.txt

@@ -1,5 +1,5 @@
 cmake_minimum_required(VERSION 3.16)
-project(proxmox-rebalancing-service VERSION 0.9.9)
+project(proxmox-rebalancing-service VERSION 1.0.2)
 
 install(PROGRAMS ../proxlb DESTINATION /bin)
 install(FILES ../proxlb.conf DESTINATION /etc/proxlb)
@@ -17,8 +17,8 @@ set(CPACK_PACKAGE_VENDOR "gyptazy")
 set(CPACK_PACKAGE_VERSION ${CMAKE_PROJECT_VERSION})
 set(CPACK_GENERATOR "RPM")
 set(CPACK_RPM_PACKAGE_ARCHITECTURE "amd64")
-set(CPACK_RPM_PACKAGE_SUMMARY "ProxLB Rebalancing VM workloads within Proxmox clusters.")
-set(CPACK_RPM_PACKAGE_DESCRIPTION "ProxLB Rebalancing VM workloads within Proxmox clusters.")
+set(CPACK_RPM_PACKAGE_SUMMARY "ProxLB - Rebalance VM workloads across nodes in Proxmox clusters.")
+set(CPACK_RPM_PACKAGE_DESCRIPTION "ProxLB - Rebalance VM workloads across nodes in Proxmox clusters.")
 set(CPACK_RPM_CHANGELOG_FILE "${CMAKE_CURRENT_SOURCE_DIR}/changelog_redhat")
 set(CPACK_PACKAGE_RELEASE 1)
 set(CPACK_RPM_PACKAGE_LICENSE "GPL 3.0")
@@ -27,15 +27,14 @@ set(CPACK_RPM_PACKAGE_REQUIRES "python >= 3.2.0")
 # DEB packaging
 set(CPACK_DEBIAN_FILE_NAME DEB-DEFAULT)
 set(CPACK_DEBIAN_PACKAGE_ARCHITECTURE "amd64")
-set(CPACK_DEBIAN_PACKAGE_SUMMARY "ProxLB Rebalancing VM workloads within Proxmox clusters.")
-set(CPACK_DEBIAN_PACKAGE_DESCRIPTION "ProxLB Rebalancing VM workloads within Proxmox clusters.")
+set(CPACK_DEBIAN_PACKAGE_SUMMARY "ProxLB - Rebalance VM workloads across nodes in Proxmox clusters.")
+set(CPACK_DEBIAN_PACKAGE_DESCRIPTION "ProxLB - Rebalance VM workloads across nodes in Proxmox clusters.")
 set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_CURRENT_SOURCE_DIR}/changelog_debian")
-set(CPACK_DEBIAN_PACKAGE_DEPENDS "python3")
+set(CPACK_DEBIAN_PACKAGE_DEPENDS "python3, python3-proxmoxer")
 set(CPACK_DEBIAN_PACKAGE_LICENSE "GPL 3.0")
 
-
 # Install
 set(CPACK_PACKAGING_INSTALL_PREFIX ${CMAKE_INSTALL_PREFIX})
-set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_CURRENT_SOURCE_DIR}/postinst")
+set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_CURRENT_SOURCE_DIR}/postinst;${CMAKE_CURRENT_SOURCE_DIR}/conffiles")
 set(CPACK_RPM_POST_INSTALL_SCRIPT_FILE "${CMAKE_CURRENT_SOURCE_DIR}/postinst")
 include(CPack)

packaging/changelog_debian

@@ -1,5 +1,21 @@
-proxlb (0.9.0) unstable; urgency=low
+proxlb (1.0.2) unstable; urgency=low
+
+  * Add option to run migration in parallel or sequentially.
+  * Add option to run ProxLB only on a Proxmox cluster master (req. HA feature).
+  * Fix daemon timer to use hours instead of minutes.
+  * Fix CMake packaging for Debian package to avoid overwriting the config file.
+  * Fix some wonkey code styles.
+
+ -- Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>  Tue, 13 Aug 2024 17:28:14 +0200
+
+proxlb (1.0.0) unstable; urgency=low
 
   * Initial release of ProxLB.
 
- -- Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>  Sun, 07 Jul 2024 05:38:41 -0200
+ -- Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>  Thu, 01 Aug 2024 17:04:12 +0200
+
+proxlb (0.9.0) unstable; urgency=low
+
+  * Initial development release of ProxLB as a tech preview.
+
+ -- Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>  Sun, 07 Jul 2024 05:38:41 +0200

packaging/changelog_redhat

@@ -1,2 +1,11 @@
-* Sun Jul 07 2024 Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>
+* Tue Aug 13 2024 Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>
+- Add option to run migration in parallel or sequentially.
+- Add option to run ProxLB only on a Proxmox cluster master (req. HA feature).
+- Fixed daemon timer to use hours instead of minutes.
+- Fixed some wonkey code styles.
+
+* Thu Aug 01 2024 Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>
 - Initial release of ProxLB.
+
+* Sun Jul 07 2024 Florian Paul Azim Hoberg <gyptazy@gyptazy.ch>
+- Initial development release of ProxLB as a tech preview.

packaging/conffiles

@@ -0,0 +1 @@
+/etc/proxlb/proxlb.conf

proxlb

@@ -33,6 +33,7 @@ except ImportError:
 import random
 import re
 import requests
+import socket
 import sys
 import time
 import urllib3
@@ -40,7 +41,7 @@ import urllib3
 
 # Constants
 __appname__ = "ProxLB"
-__version__ = "0.9.9"
+__version__ = "1.0.2"
 __author__  = "Florian Paul Azim Hoberg <gyptazy@gyptazy.ch> @gyptazy"
 __errors__ = False
 
@@ -112,7 +113,7 @@ def validate_daemon(daemon, schedule):
 
     if bool(int(daemon)):
         logging.info(f'{info_prefix} Running in daemon mode. Next run in {schedule} hours.')
-        time.sleep(int(schedule) * 60)
+        time.sleep(int(schedule) * 60 * 60)
     else:
         logging.info(f'{info_prefix} Not running in daemon mode. Quitting.')
         sys.exit(0)
@@ -145,7 +146,7 @@ def __validate_config_file(config_path):
 def initialize_args():
     """ Initialize given arguments for ProxLB. """
     argparser = argparse.ArgumentParser(description='ProxLB')
-    argparser.add_argument('-c', '--config',  type=str, help='Path to config file.', required=True)
+    argparser.add_argument('-c', '--config',  type=str, help='Path to config file.', required=False)
     argparser.add_argument('-d', '--dry-run', help='Perform a dry-run without doing any actions.', action='store_true', required=False)
     argparser.add_argument('-j', '--json',    help='Return a JSON of the VM movement.', action='store_true', required=False)
     return argparser.parse_args()
@@ -173,21 +174,24 @@ def initialize_config_options(config_path):
         config = configparser.ConfigParser()
         config.read(config_path)
         # Proxmox config
-        proxmox_api_host  = config['proxmox']['api_host']
-        proxmox_api_user  = config['proxmox']['api_user']
-        proxmox_api_pass  = config['proxmox']['api_pass']
-        proxmox_api_ssl_v = config['proxmox']['verify_ssl']
+        proxmox_api_host      = config['proxmox']['api_host']
+        proxmox_api_user      = config['proxmox']['api_user']
+        proxmox_api_pass      = config['proxmox']['api_pass']
+        proxmox_api_ssl_v     = config['proxmox']['verify_ssl']
         # Balancing
-        balancing_method  = config['balancing'].get('method', 'memory')
-        balancing_mode    = config['balancing'].get('mode', 'used')
-        balancing_type    = config['balancing'].get('type', 'vm')
-        balanciness       = config['balancing'].get('balanciness', 10)
-        ignore_nodes      = config['balancing'].get('ignore_nodes', None)
-        ignore_vms        = config['balancing'].get('ignore_vms', None)
+        balancing_method      = config['balancing'].get('method', 'memory')
+        balancing_mode        = config['balancing'].get('mode', 'used')
+        balancing_mode_option = config['balancing'].get('mode_option', 'bytes')
+        balancing_type        = config['balancing'].get('type', 'vm')
+        balanciness           = config['balancing'].get('balanciness', 10)
+        parallel_migrations   = config['balancing'].get('parallel_migrations', 1)
+        ignore_nodes          = config['balancing'].get('ignore_nodes', None)
+        ignore_vms            = config['balancing'].get('ignore_vms', None)
         # Service
-        daemon            = config['service'].get('daemon', 1)
-        schedule          = config['service'].get('schedule', 24)
-        log_verbosity     = config['service'].get('log_verbosity', 'CRITICAL')
+        master_only           = config['service'].get('master_only', 0)
+        daemon                = config['service'].get('daemon', 1)
+        schedule              = config['service'].get('schedule', 24)
+        log_verbosity         = config['service'].get('log_verbosity', 'CRITICAL')
     except configparser.NoSectionError:
         logging.critical(f'{error_prefix} Could not find the required section.')
         sys.exit(2)
@@ -199,8 +203,8 @@ def initialize_config_options(config_path):
         sys.exit(2)
 
     logging.info(f'{info_prefix} Configuration file loaded.')
-    return proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v, balancing_method, \
-         balancing_mode, balancing_type, balanciness, ignore_nodes, ignore_vms, daemon, schedule, log_verbosity
+    return proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v, balancing_method, balancing_mode, balancing_mode_option, \
+         balancing_type, balanciness, parallel_migrations, ignore_nodes, ignore_vms, master_only, daemon, schedule, log_verbosity
 
 
 def api_connect(proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v):
@@ -230,6 +234,62 @@ def api_connect(proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_ap
     return api_object
 
 
+def execute_rebalancing_only_by_master(api_object, master_only):
+    """ Validate if balancing should only be done by the cluster master. Afterwards, validate if this node is the cluster master. """
+    info_prefix  = 'Info: [only-on-master-executor]:'
+    master_only = bool(int(master_only))
+
+    if bool(int(master_only)):
+        logging.info(f'{info_prefix} Master only rebalancing is defined. Starting validation.')
+        cluster_master_node = get_cluster_master(api_object)
+        cluster_master = validate_cluster_master(cluster_master_node)
+        return cluster_master, master_only
+    else:
+        logging.info(f'{info_prefix} No master only rebalancing is defined. Skipping validation.')
+        return False, master_only
+
+
+def get_cluster_master(api_object):
+    """ Get the current master of the Proxmox cluster. """
+    error_prefix = 'Error: [cluster-master-getter]:'
+    info_prefix  = 'Info: [cluster-master-getter]:'
+
+    try:
+        ha_status_object = api_object.cluster().ha().status().manager_status().get()
+        logging.info(f'{info_prefix} Master node: {ha_status_object.get("manager_status", None).get("master_node", None)}')
+    except urllib3.exceptions.NameResolutionError:
+        logging.critical(f'{error_prefix} Could not resolve the API.')
+        sys.exit(2)
+    except requests.exceptions.ConnectTimeout:
+        logging.critical(f'{error_prefix} Connection time out to API.')
+        sys.exit(2)
+    except requests.exceptions.SSLError:
+        logging.critical(f'{error_prefix} SSL certificate verification failed for API.')
+        sys.exit(2)
+
+    cluster_master = ha_status_object.get("manager_status", None).get("master_node", None)
+
+    if cluster_master:
+        return cluster_master
+    else:
+        logging.critical(f'{error_prefix} Could not obtain cluster master. Please check your configuration - stopping.')
+        sys.exit(2)
+
+
+def validate_cluster_master(cluster_master):
+    """ Validate if the current execution node is the cluster master. """
+    info_prefix  = 'Info: [cluster-master-validator]:'
+
+    node_executor_hostname = socket.gethostname()
+    logging.info(f'{info_prefix} Node executor hostname is: {node_executor_hostname}')
+
+    if node_executor_hostname != cluster_master:
+        logging.info(f'{info_prefix} {node_executor_hostname} is not the cluster master ({cluster_master}).')
+        return False
+    else:
+        return True
+
+
 def get_node_statistics(api_object, ignore_nodes):
     """ Get statistics of cpu, memory and disk for each node in the cluster. """
     info_prefix       = 'Info: [node-statistics]:'
@@ -444,26 +504,27 @@ def __get_proxlb_groups(vm_tags):
     return group_include, group_exclude, vm_ignore
 
 
-def balancing_calculations(balancing_method, balancing_mode, node_statistics, vm_statistics, balanciness, rebalance, processed_vms):
+def balancing_calculations(balancing_method, balancing_mode, balancing_mode_option, node_statistics, vm_statistics, balanciness, rebalance, processed_vms):
     """ Calculate re-balancing of VMs on present nodes across the cluster. """
     info_prefix  = 'Info: [rebalancing-calculator]:'
 
     # Validate for a supported balancing method, mode and if rebalancing is required.
     __validate_balancing_method(balancing_method)
     __validate_balancing_mode(balancing_mode)
+    __validate_vm_statistics(vm_statistics)
     rebalance = __validate_balanciness(balanciness, balancing_method, balancing_mode, node_statistics)
 
     if rebalance:
         # Get most used/assigned resources of the VM and the most free or less allocated node.
         resources_vm_most_used, processed_vms  = __get_most_used_resources_vm(balancing_method, balancing_mode, vm_statistics, processed_vms)
-        resources_node_most_free               = __get_most_free_resources_node(balancing_method, balancing_mode, node_statistics)
+        resources_node_most_free               = __get_most_free_resources_node(balancing_method, balancing_mode, balancing_mode_option, node_statistics)
 
         # Update resource statistics for VMs and nodes.
         node_statistics, vm_statistics         = __update_resource_statistics(resources_vm_most_used, resources_node_most_free,
                                                                              vm_statistics, node_statistics, balancing_method, balancing_mode)
 
         # Start recursion until we do not have any needs to rebalance anymore.
-        balancing_calculations(balancing_method, balancing_mode, node_statistics, vm_statistics, balanciness, rebalance, processed_vms)
+        balancing_calculations(balancing_method, balancing_mode, balancing_mode_option, node_statistics, vm_statistics, balanciness, rebalance, processed_vms)
 
     # Honour groupings for include and exclude groups for rebalancing VMs.
     node_statistics, vm_statistics = __get_vm_tags_include_groups(vm_statistics, node_statistics, balancing_method, balancing_mode)
@@ -481,7 +542,7 @@ def balancing_calculations(balancing_method, balancing_mode, node_statistics, vm
 def __validate_balancing_method(balancing_method):
     """ Validate for valid and supported balancing method. """
     error_prefix = 'Error: [balancing-method-validation]:'
-    info_prefix  = 'Info: [balancing-method-validation]]:'
+    info_prefix  = 'Info: [balancing-method-validation]:'
 
     if balancing_method not in ['memory', 'disk', 'cpu']:
         logging.error(f'{error_prefix} Invalid balancing method: {balancing_method}')
@@ -493,7 +554,7 @@ def __validate_balancing_method(balancing_method):
 def __validate_balancing_mode(balancing_mode):
     """ Validate for valid and supported balancing mode. """
     error_prefix = 'Error: [balancing-mode-validation]:'
-    info_prefix  = 'Info: [balancing-mode-validation]]:'
+    info_prefix  = 'Info: [balancing-mode-validation]:'
 
     if balancing_mode not in ['used', 'assigned']:
         logging.error(f'{error_prefix} Invalid balancing method: {balancing_mode}')
@@ -502,6 +563,15 @@ def __validate_balancing_mode(balancing_mode):
         logging.info(f'{info_prefix} Valid balancing method: {balancing_mode}')
 
 
+def __validate_vm_statistics(vm_statistics):
+    """ Validate for at least a single object of type CT/VM to rebalance. """
+    error_prefix = 'Error: [balancing-vm-stats-validation]:'
+
+    if len(vm_statistics) == 0:
+        logging.error(f'{error_prefix} Not a single CT/VM found in cluster.')
+        sys.exit(1)
+
+
 def __validate_balanciness(balanciness, balancing_method, balancing_mode, node_statistics):
     """ Validate for balanciness to ensure further rebalancing is needed. """
     info_prefix                   = 'Info: [balanciness-validation]:'
@@ -566,13 +636,15 @@ def __get_most_used_resources_vm(balancing_method, balancing_mode, vm_statistics
     return vm, processed_vms
 
 
-def __get_most_free_resources_node(balancing_method, balancing_mode, node_statistics):
+def __get_most_free_resources_node(balancing_method, balancing_mode, balancing_mode_option, node_statistics):
     """ Get and return the most free resources of a node by the defined balancing method. """
     info_prefix = 'Info: [get-most-free-resources-nodes]:'
 
     # Return the node information based on the balancing mode.
-    if balancing_mode == 'used':
+    if balancing_mode == 'used' and balancing_mode_option == 'bytes':
         node = max(node_statistics.items(), key=lambda item: item[1][f'{balancing_method}_free'])
+    if balancing_mode == 'used' and balancing_mode_option == 'percent':
+        node = max(node_statistics.items(), key=lambda item: item[1][f'{balancing_method}_free_percent'])
     if balancing_mode == 'assigned':
         node = min(node_statistics.items(), key=lambda item: item[1][f'{balancing_method}_assigned'] if item[1][f'{balancing_method}_assigned_percent'] > 0 or item[1][f'{balancing_method}_assigned_percent'] < 100 else -float('inf'))
 
@@ -690,18 +762,59 @@ def __get_vm_tags_exclude_groups(vm_statistics, node_statistics, balancing_metho
     return node_statistics, vm_statistics
 
 
-def __run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args):
+def __wait_job_finalized(api_object, node_name, job_id, counter):
+    """ Wait for a job to be finalized. """
+    error_prefix = 'Error: [job-status-getter]:'
+    info_prefix  = 'Info: [job-status-getter]:'
+
+    logging.info(f'{info_prefix} Getting job status for job {job_id}.')
+    task = api_object.nodes(node_name).tasks(job_id).status().get()
+    logging.info(f'{info_prefix} {task}')
+
+    if task['status'] == 'running':
+        logging.info(f'{info_prefix} Validating job {job_id} for the {counter} run.')
+
+        # Do not run for infinity this recursion and fail when reaching the limit.
+        if counter == 300:
+            logging.critical(f'{error_prefix} The job {job_id} on node {node_name} did not finished in time for migration.')
+
+        time.sleep(5)
+        counter = counter + 1
+        logging.info(f'{info_prefix} Revalidating job {job_id} in a next run.')
+        __wait_job_finalized(api_object, node_name, job_id, counter)
+
+    logging.info(f'{info_prefix} Job {job_id} for migration from {node_name} terminiated succesfully.')
+
+
+def __run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args, parallel_migrations):
     """ Run & execute the VM rebalancing via API. """
     error_prefix = 'Error: [rebalancing-executor]:'
     info_prefix  = 'Info: [rebalancing-executor]:'
 
     if len(vm_statistics_rebalanced) > 0 and not app_args.dry_run:
         for vm, value in vm_statistics_rebalanced.items():
+
             try:
-                logging.info(f'{info_prefix} Rebalancing vm {vm} from node {value["node_parent"]} to node {value["node_rebalance"]}.')
-                api_object.nodes(value['node_parent']).qemu(value['vmid']).migrate().post(target=value['node_rebalance'],online=1)
+                # Migrate type VM (live migration).
+                if value['type'] == 'vm':
+                    logging.info(f'{info_prefix} Rebalancing VM {vm} from node {value["node_parent"]} to node {value["node_rebalance"]}.')
+                    job_id = api_object.nodes(value['node_parent']).qemu(value['vmid']).migrate().post(target=value['node_rebalance'],online=1)
+
+                # Migrate type CT (requires restart of container).
+                if value['type'] == 'ct':
+                    logging.info(f'{info_prefix} Rebalancing CT {vm} from node {value["node_parent"]} to node {value["node_rebalance"]}.')
+                    job_id = api_object.nodes(value['node_parent']).lxc(value['vmid']).migrate().post(target=value['node_rebalance'],restart=1)
+
             except proxmoxer.core.ResourceException as error_resource:
                 logging.critical(f'{error_prefix} {error_resource}')
+
+            # Wait for migration to be finished unless running parallel migrations.
+            if not bool(int(parallel_migrations)):
+                logging.info(f'{info_prefix} Rebalancing will be performed sequentially.')
+                __wait_job_finalized(api_object, value['node_parent'], job_id, counter=1)
+            else:
+                logging.info(f'{info_prefix} Rebalancing will be performed parallely.')
+
     else:
         logging.info(f'{info_prefix} No rebalancing needed.')
 
@@ -715,40 +828,58 @@ def __create_json_output(vm_statistics_rebalanced, app_args):
         print(json.dumps(vm_statistics_rebalanced))
 
 
-def __create_dry_run_output(vm_statistics_rebalanced, app_args):
+def __create_cli_output(vm_statistics_rebalanced, app_args):
     """ Create output for CLI when running in dry-run mode. """
-    info_prefix  = 'Info: [dry-run-output-generator]:'
-    vm_to_node_list = []
+    info_prefix_dry_run  = 'Info: [cli-output-generator-dry-run]:'
+    info_prefix_run      = 'Info: [cli-output-generator]:'
+    vm_to_node_list      = []
+
+    if app_args.dry_run:
+        info_prefix = info_prefix_dry_run
+        logging.info(f'{info_prefix} Starting dry-run to rebalance vms to their new nodes.')
+    else:
+        info_prefix = info_prefix_run
+        logging.info(f'{info_prefix} Start rebalancing vms to their new nodes.')
 
-    logging.info(f'{info_prefix} Starting dry-run to rebalance vms to their new nodes.')
     vm_to_node_list.append(['VM', 'Current Node', 'Rebalanced Node', 'VM Type'])
     for vm_name, vm_values in vm_statistics_rebalanced.items():
         vm_to_node_list.append([vm_name, vm_values['node_parent'], vm_values['node_rebalance'], vm_values['type']])
 
     if len(vm_statistics_rebalanced) > 0:
         logging.info(f'{info_prefix} Printing cli output of VM rebalancing.')
-        __print_table_cli(vm_to_node_list)
+        __print_table_cli(vm_to_node_list, app_args.dry_run)
     else:
         logging.info(f'{info_prefix} No rebalancing needed.')
 
 
-def __print_table_cli(table):
+def __print_table_cli(table, dry_run=False):
     """ Pretty print a given table to the cli. """
+    info_prefix_dry_run  = 'Info: [cli-output-generator-table-dryn-run]:'
+    info_prefix_run      = 'Info: [cli-output-generator-table]:'
+    info_prefix          = info_prefix_run
+
     longest_cols = [
         (max([len(str(row[i])) for row in table]) + 3)
         for i in range(len(table[0]))
     ]
 
     row_format = "".join(["{:>" + str(longest_col) + "}" for longest_col in longest_cols])
+
     for row in table:
-        print(row_format.format(*row))
+        # Print CLI output when running in dry-run mode to make the user's life easier.
+        if dry_run:
+            info_prefix = info_prefix_dry_run
+            print(row_format.format(*row))
+
+        # Log all items in info mode.
+        logging.info(f'{info_prefix} {row_format.format(*row)}')
 
 
-def run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args):
+def run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args, parallel_migrations):
     """ Run rebalancing of vms to new nodes in cluster. """
-    __run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args)
+    __run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args, parallel_migrations)
     __create_json_output(vm_statistics_rebalanced, app_args)
-    __create_dry_run_output(vm_statistics_rebalanced, app_args)
+    __create_cli_output(vm_statistics_rebalanced, app_args)
 
 
 def main():
@@ -760,8 +891,8 @@ def main():
     pre_validations(config_path)
 
     # Parse global config.
-    proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v, balancing_method, balancing_mode, balancing_type, \
-        balanciness, ignore_nodes, ignore_vms, daemon, schedule, log_verbosity = initialize_config_options(config_path)
+    proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v, balancing_method, balancing_mode, balancing_mode_option, balancing_type, \
+        balanciness, parallel_migrations, ignore_nodes, ignore_vms, master_only, daemon, schedule, log_verbosity = initialize_config_options(config_path)
 
     # Overwrite logging handler with user defined log verbosity.
     initialize_logger(log_verbosity, update_log_verbosity=True)
@@ -770,16 +901,26 @@ def main():
         # API Authentication.
         api_object = api_connect(proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v)
 
+        # Get master node of cluster and ensure that ProxLB is only performed on the
+        # cluster master node to avoid ongoing rebalancing.
+        cluster_master, master_only = execute_rebalancing_only_by_master(api_object, master_only)
+
+        # Validate daemon service and skip following tasks when not being the cluster master.
+        if not cluster_master and master_only:
+            validate_daemon(daemon, schedule)
+            continue
+
         # Get metric & statistics for vms and nodes.
         node_statistics = get_node_statistics(api_object, ignore_nodes)
         vm_statistics   = get_vm_statistics(api_object, ignore_vms, balancing_type)
         node_statistics = update_node_statistics(node_statistics, vm_statistics)
 
         # Calculate rebalancing of vms.
-        node_statistics_rebalanced, vm_statistics_rebalanced = balancing_calculations(balancing_method, balancing_mode, node_statistics, vm_statistics, balanciness, rebalance=False, processed_vms=[])
+        node_statistics_rebalanced, vm_statistics_rebalanced = balancing_calculations(balancing_method, balancing_mode, balancing_mode_option,
+                                                                                      node_statistics, vm_statistics, balanciness, rebalance=False, processed_vms=[])
 
         # Rebalance vms to new nodes within the cluster.
-        run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args)
+        run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args, parallel_migrations)
 
         # Validate for any errors.
         post_validations()