github/ProxLB
1
1
Fork 0
mirror of https://github.com/gyptazy/ProxLB.git synced 2026-04-05 20:31:57 +02:00
Code Issues Packages Projects Releases Wiki Activity

feature: Add cli arg (-b) to return the best next node for VM placement.

Browse Source 
Fixes: #8
Fixes: #53
This commit is contained in:
Florian Paul Azim Hoberg
2024-08-19 20:57:29 +02:00
parent 143135f1d8
commit 39142780d5
7 changed files with 154 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.changelogs/1.0.3/53_code_improvements.yml Normal file
View File

@@ -0,0 +1,6 @@
added:
- Added convert function to cast all bool alike options from configparser to bools. [#53]
- Added config parser options for future features. [#53]
- Added a config versio schema that must be supported by ProxLB. [#53]
changed:
- Improved the underlying code base for future implementations. [#53]

2
.changelogs/1.0.3/8_add_best_next_node_for_placement.yml Normal file
View File

@@ -0,0 +1,2 @@
added:
- Added cli arg `-b` to return the next best node for next VM/CT placement. [#8]

2
.changelogs/1.0.3/bug_fix_cluster_master_only.yml Normal file
View File

@@ -0,0 +1,2 @@
fixed:
- Fixed `master_only` function by inverting the condition.

1
.changelogs/1.0.3/release_meta.yml Normal file
View File

@@ -0,0 +1 @@
date: TBD

50
README.md
View File

@@ -98,24 +98,29 @@ Running PLB is easy and it runs almost everywhere since it just depends on `Pyth
### Options
The following options can be set in the `proxlb.conf` file:
| Option | Example | Description |
|------|:------:|:------:|
| api_host | hypervisor01.gyptazy.ch | Host or IP address of the remote Proxmox API. |
| api_user | root@pam | Username for the API. |
| api_pass | FooBar | Password for the API. |
| verify_ssl | 1 | Validate SSL certificates (1) or ignore (0). (default: 1) |
| method | memory | Defines the balancing method (default: memory) where you can use `memory`, `disk` or `cpu`. |
| mode | used | Rebalance by `used` resources (efficiency) or `assigned` (avoid overprovisioning) resources. (default: used)|
| mode_option | byte | Rebalance by node's resources in `bytes` or `percent`. (default: bytes) |
| type | vm | Rebalance only `vm` (virtual machines), `ct` (containers) or `all` (virtual machines & containers). (default: vm)|
| balanciness | 10 | Value of the percentage of lowest and highest resource consumption on nodes may differ before rebalancing. (default: 10) |
| parallel_migrations | 1 | Defines if migrations should be done parallely or sequentially. (default: 1) |
| ignore_nodes | dummynode01,dummynode02,test* | Defines a comma separated list of nodes to exclude. |
| ignore_vms | testvm01,testvm02 | Defines a comma separated list of VMs to exclude. (`*` as suffix wildcard or tags are also supported) |
| master_only | 0 | Defines is this should only be performed (1) on the cluster master node or not (0). (default: 0) |
| daemon | 1 | Run as a daemon (1) or one-shot (0). (default: 1) |
| schedule | 24 | Hours to rebalance in hours. (default: 24) |
| log_verbosity | INFO | Defines the log level (default: CRITICAL) where you can use `INFO`, `WARN` or `CRITICAL` |
| Section | Option | Example | Description |
|------|:------:|:------:|:------:|
| `proxmox` | api_host | hypervisor01.gyptazy.ch | Host or IP address of the remote Proxmox API. |
| | api_user | root@pam | Username for the API. |
| | api_pass | FooBar | Password for the API. |
| | verify_ssl | 1 | Validate SSL certificates (1) or ignore (0). (default: 1) |
| `vm_balancing` | enable | 1 | Enables VM/CT balancing. |
| | method | memory | Defines the balancing method (default: memory) where you can use `memory`, `disk` or `cpu`. |
| | mode | used | Rebalance by `used` resources (efficiency) or `assigned` (avoid overprovisioning) resources. (default: used)|
| | mode_option | byte | Rebalance by node's resources in `bytes` or `percent`. (default: bytes) |
| | type | vm | Rebalance only `vm` (virtual machines), `ct` (containers) or `all` (virtual machines & containers). (default: vm)|
| | balanciness | 10 | Value of the percentage of lowest and highest resource consumption on nodes may differ before rebalancing. (default: 10) |
| | parallel_migrations | 1 | Defines if migrations should be done parallely or sequentially. (default: 1) |
| | ignore_nodes | dummynode01,dummynode02,test* | Defines a comma separated list of nodes to exclude. |
| | ignore_vms | testvm01,testvm02 | Defines a comma separated list of VMs to exclude. (`*` as suffix wildcard or tags are also supported) |
| | master_only | 0 | Defines is this should only be performed (1) on the cluster master node or not (0). (default: 0) |
| `storage_balancing` | enable | 0 | Enables storage balancing. |
| `update_service` | enable | 0 | Enables the automated update service (rolling updates). |
| `api` | enable | 0 | Enables the ProxLB API. |
| | daemon | 1 | Run as a daemon (1) or one-shot (0). (default: 1) |
| | schedule | 24 | Hours to rebalance in hours. (default: 24) |
| | log_verbosity | INFO | Defines the log level (default: CRITICAL) where you can use `INFO`, `WARN` or `CRITICAL` |
| | config_version | 3 | Defines the current config version schema for ProxLB |
An example of the configuration file looks like:
```
@@ -124,7 +129,8 @@ api_host: hypervisor01.gyptazy.ch
api_user: root@pam
api_pass: FooBar
verify_ssl: 1
[balancing]
[vm_balancing]
enable: 1
method: memory
mode: used
type: vm
@@ -146,6 +152,7 @@ ignore_vms: testvm01,testvm02
# HA status.
master_only: 0
daemon: 1
config_version: 3
```
### Parameters
@@ -154,8 +161,9 @@ The following options and parameters are currently supported:
| Option | Long Option | Description | Default |
|------|:------:|------:|------:|
| -c | --config | Path to a config file. | /etc/proxlb/proxlb.conf (default) |
| -d | --dry-run | Perform a dry-run without doing any actions. | Unset |
| -j | --json | Return a JSON of the VM movement. | Unset |
| -d | --dry-run | Performs a dry-run without doing any actions. | Unset |
| -j | --json | Returns a JSON of the VM movement. | Unset |
| -b | --best-node | Returns the best next node for a VM/CT placement (useful for further usage with Terraform/Ansible). | Unset |
### Balancing
#### General

157
proxlb
View File

@@ -40,10 +40,11 @@ import urllib3
# Constants
__appname__ = "ProxLB"
__version__ = "1.0.2"
__author__ = "Florian Paul Azim Hoberg <gyptazy@gyptazy.ch> @gyptazy"
__errors__ = False
__appname__ = "ProxLB"
__version__ = "1.0.3b"
__config_version__ = 3
__author__ = "Florian Paul Azim Hoberg <gyptazy@gyptazy.ch> @gyptazy"
__errors__ = False
# Classes
@@ -146,9 +147,10 @@ def __validate_config_file(config_path):
def initialize_args():
""" Initialize given arguments for ProxLB. """
argparser = argparse.ArgumentParser(description='ProxLB')
argparser.add_argument('-c', '--config', type=str, help='Path to config file.', required=False)
argparser.add_argument('-d', '--dry-run', help='Perform a dry-run without doing any actions.', action='store_true', required=False)
argparser.add_argument('-j', '--json', help='Return a JSON of the VM movement.', action='store_true', required=False)
argparser.add_argument('-c', '--config', type=str, help='Path to config file.', required=False)
argparser.add_argument('-d', '--dry-run', help='Perform a dry-run without doing any actions.', action='store_true', required=False)
argparser.add_argument('-j', '--json', help='Return a JSON of the VM movement.', action='store_true', required=False)
argparser.add_argument('-b', '--best-node', help='Returns the best next node.', action='store_true', required=False)
return argparser.parse_args()
@@ -167,31 +169,40 @@ def initialize_config_path(app_args):
def initialize_config_options(config_path):
""" Read configuration from given config file for ProxLB. """
error_prefix = 'Error: [config]:'
info_prefix = 'Info: [config]:'
error_prefix = 'Error: [config]:'
info_prefix = 'Info: [config]:'
proxlb_config = {}
try:
config = configparser.ConfigParser()
config.read(config_path)
# Proxmox config
proxmox_api_host = config['proxmox']['api_host']
proxmox_api_user = config['proxmox']['api_user']
proxmox_api_pass = config['proxmox']['api_pass']
proxmox_api_ssl_v = config['proxmox']['verify_ssl']
# Balancing
balancing_method = config['balancing'].get('method', 'memory')
balancing_mode = config['balancing'].get('mode', 'used')
balancing_mode_option = config['balancing'].get('mode_option', 'bytes')
balancing_type = config['balancing'].get('type', 'vm')
balanciness = config['balancing'].get('balanciness', 10)
parallel_migrations = config['balancing'].get('parallel_migrations', 1)
ignore_nodes = config['balancing'].get('ignore_nodes', None)
ignore_vms = config['balancing'].get('ignore_vms', None)
proxlb_config['proxmox_api_host'] = config['proxmox']['api_host']
proxlb_config['proxmox_api_user'] = config['proxmox']['api_user']
proxlb_config['proxmox_api_pass'] = config['proxmox']['api_pass']
proxlb_config['proxmox_api_ssl_v'] = config['proxmox']['verify_ssl']
# VM Balancing
proxlb_config['vm_balancing_enable'] = config['vm_balancing'].get('enable', 1)
proxlb_config['vm_balancing_method'] = config['vm_balancing'].get('method', 'memory')
proxlb_config['vm_balancing_mode'] = config['vm_balancing'].get('mode', 'used')
proxlb_config['vm_balancing_mode_option'] = config['vm_balancing'].get('mode_option', 'bytes')
proxlb_config['vm_balancing_type'] = config['vm_balancing'].get('type', 'vm')
proxlb_config['vm_balanciness'] = config['vm_balancing'].get('balanciness', 10)
proxlb_config['vm_parallel_migrations'] = config['vm_balancing'].get('parallel_migrations', 1)
proxlb_config['vm_ignore_nodes'] = config['vm_balancing'].get('ignore_nodes', None)
proxlb_config['vm_ignore_vms'] = config['vm_balancing'].get('ignore_vms', None)
# Storage Balancing
proxlb_config['storage_balancing_enable'] = config['storage_balancing'].get('enable', 0)
# Update Support
proxlb_config['update_service'] = config['update_service'].get('enable', 0)
# API
proxlb_config['api'] = config['update_service'].get('enable', 0)
# Service
master_only = config['service'].get('master_only', 0)
daemon = config['service'].get('daemon', 1)
schedule = config['service'].get('schedule', 24)
log_verbosity = config['service'].get('log_verbosity', 'CRITICAL')
proxlb_config['master_only'] = config['service'].get('master_only', 0)
proxlb_config['daemon'] = config['service'].get('daemon', 1)
proxlb_config['schedule'] = config['service'].get('schedule', 24)
proxlb_config['log_verbosity'] = config['service'].get('log_verbosity', 'CRITICAL')
proxlb_config['config_version'] = config['service'].get('config_version', 2)
except configparser.NoSectionError:
logging.critical(f'{error_prefix} Could not find the required section.')
sys.exit(2)
@@ -202,9 +213,43 @@ def initialize_config_options(config_path):
logging.critical(f'{error_prefix} Could not find the required options in config file.')
sys.exit(2)
# Normalize and update bools. Afterwards, validate minimum required config version.
proxlb_config = __update_config_parser_bools(proxlb_config)
validate_config_minimum_version(proxlb_config)
logging.info(f'{info_prefix} Configuration file loaded.')
return proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v, balancing_method, balancing_mode, balancing_mode_option, \
balancing_type, balanciness, parallel_migrations, ignore_nodes, ignore_vms, master_only, daemon, schedule, log_verbosity
return proxlb_config
def __update_config_parser_bools(proxlb_config):
""" Update bools in config from configparser to real bools """
info_prefix = 'Info: [config-bool-converter]:'
# Normalize and update config parser values to bools.
for section, option_value in proxlb_config.items():
if option_value in [1, '1', 'yes', 'Yes', 'true', 'True', 'enable']:
logging.info(f'{info_prefix} Converting {section} to bool: True.')
proxlb_config[section] = True
if option_value in [0, '0', 'no', 'No', 'false', 'False', 'disable']:
logging.info(f'{info_prefix} Converting {section} to bool: False.')
proxlb_config[section] = False
return proxlb_config
def validate_config_minimum_version(proxlb_config):
""" Validate the minimum required config file for ProxLB """
info_prefix = 'Info: [config-version-validator]:'
error_prefix = 'Error: [config-version-validator]:'
if int(proxlb_config['config_version']) < __config_version__:
logging.error(f'{error_prefix} ProxLB config version {proxlb_config["config_version"]} is too low. Required: {__config_version__}.')
print(f'{error_prefix} ProxLB config version {proxlb_config["config_version"]} is too low. Required: {__config_version__}.')
sys.exit(1)
else:
logging.info(f'{info_prefix} ProxLB config version {proxlb_config["config_version"]} is fine. Required: {__config_version__}.')
def api_connect(proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v):
@@ -504,9 +549,9 @@ def __get_proxlb_groups(vm_tags):
return group_include, group_exclude, vm_ignore
def balancing_calculations(balancing_method, balancing_mode, balancing_mode_option, node_statistics, vm_statistics, balanciness, rebalance, processed_vms):
def balancing_vm_calculations(balancing_method, balancing_mode, balancing_mode_option, node_statistics, vm_statistics, balanciness, app_args, rebalance, processed_vms):
""" Calculate re-balancing of VMs on present nodes across the cluster. """
info_prefix = 'Info: [rebalancing-calculator]:'
info_prefix = 'Info: [rebalancing-vm-calculator]:'
# Validate for a supported balancing method, mode and if rebalancing is required.
__validate_balancing_method(balancing_method)
@@ -520,11 +565,20 @@ def balancing_calculations(balancing_method, balancing_mode, balancing_mode_opti
resources_node_most_free = __get_most_free_resources_node(balancing_method, balancing_mode, balancing_mode_option, node_statistics)
# Update resource statistics for VMs and nodes.
node_statistics, vm_statistics = __update_resource_statistics(resources_vm_most_used, resources_node_most_free,
node_statistics, vm_statistics = __update_vm_resource_statistics(resources_vm_most_used, resources_node_most_free,
vm_statistics, node_statistics, balancing_method, balancing_mode)
# Start recursion until we do not have any needs to rebalance anymore.
balancing_calculations(balancing_method, balancing_mode, balancing_mode_option, node_statistics, vm_statistics, balanciness, rebalance, processed_vms)
balancing_vm_calculations(balancing_method, balancing_mode, balancing_mode_option, node_statistics, vm_statistics, balanciness, app_args, rebalance, processed_vms)
# If only best node argument set we simply return the next best node for VM
# and CT placement on the CLI and stop ProxLB.
if app_args.best_node:
logging.info(f'{info_prefix} Only best next node for new VM & CT placement requsted.')
best_next_node = __get_most_free_resources_node(balancing_method, balancing_mode, balancing_mode_option, node_statistics)
print(best_next_node[0])
logging.info(f'{info_prefix} Best next node for VM & CT placement: {best_next_node[0]}')
sys.exit(0)
# Honour groupings for include and exclude groups for rebalancing VMs.
node_statistics, vm_statistics = __get_vm_tags_include_groups(vm_statistics, node_statistics, balancing_method, balancing_mode)
@@ -652,7 +706,7 @@ def __get_most_free_resources_node(balancing_method, balancing_mode, balancing_m
return node
def __update_resource_statistics(resource_highest_used_resources_vm, resource_highest_free_resources_node, vm_statistics, node_statistics, balancing_method, balancing_mode):
def __update_vm_resource_statistics(resource_highest_used_resources_vm, resource_highest_free_resources_node, vm_statistics, node_statistics, balancing_method, balancing_mode):
""" Update VM and node resource statistics. """
info_prefix = 'Info: [rebalancing-resource-statistics-update]:'
@@ -717,7 +771,7 @@ def __get_vm_tags_include_groups(vm_statistics, node_statistics, balancing_metho
vm_node_rebalance = vm_statistics[vm_name]['node_rebalance']
else:
_mocked_vm_object = (vm_name, vm_statistics[vm_name])
node_statistics, vm_statistics = __update_resource_statistics(_mocked_vm_object, [vm_node_rebalance], vm_statistics, node_statistics, balancing_method, balancing_mode)
node_statistics, vm_statistics = __update_vm_resource_statistics(_mocked_vm_object, [vm_node_rebalance], vm_statistics, node_statistics, balancing_method, balancing_mode)
processed_vm.append(vm_name)
return node_statistics, vm_statistics
@@ -756,7 +810,7 @@ def __get_vm_tags_exclude_groups(vm_statistics, node_statistics, balancing_metho
random_node = random.choice(list(node_statistics.keys()))
else:
_mocked_vm_object = (vm_name, vm_statistics[vm_name])
node_statistics, vm_statistics = __update_resource_statistics(_mocked_vm_object, [random_node], vm_statistics, node_statistics, balancing_method, balancing_mode)
node_statistics, vm_statistics = __update_vm_resource_statistics(_mocked_vm_object, [random_node], vm_statistics, node_statistics, balancing_method, balancing_mode)
processed_vm.append(vm_name)
return node_statistics, vm_statistics
@@ -891,42 +945,43 @@ def main():
pre_validations(config_path)
# Parse global config.
proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v, balancing_method, balancing_mode, balancing_mode_option, balancing_type, \
balanciness, parallel_migrations, ignore_nodes, ignore_vms, master_only, daemon, schedule, log_verbosity = initialize_config_options(config_path)
proxlb_config = initialize_config_options(config_path)
# Overwrite logging handler with user defined log verbosity.
initialize_logger(log_verbosity, update_log_verbosity=True)
initialize_logger(proxlb_config['log_verbosity'], update_log_verbosity=True)
while True:
# API Authentication.
api_object = api_connect(proxmox_api_host, proxmox_api_user, proxmox_api_pass, proxmox_api_ssl_v)
api_object = api_connect(proxlb_config['proxmox_api_host'], proxlb_config['proxmox_api_user'], proxlb_config['proxmox_api_pass'], proxlb_config['proxmox_api_ssl_v'])
# Get master node of cluster and ensure that ProxLB is only performed on the
# cluster master node to avoid ongoing rebalancing.
cluster_master, master_only = execute_rebalancing_only_by_master(api_object, master_only)
cluster_master, master_only = execute_rebalancing_only_by_master(api_object, proxlb_config['master_only'])
# Validate daemon service and skip following tasks when not being the cluster master.
if not cluster_master and master_only:
validate_daemon(daemon, schedule)
validate_daemon(proxlb_config['daemon'], proxlb_config['schedule'])
continue
# Get metric & statistics for vms and nodes.
node_statistics = get_node_statistics(api_object, ignore_nodes)
vm_statistics = get_vm_statistics(api_object, ignore_vms, balancing_type)
node_statistics = update_node_statistics(node_statistics, vm_statistics)
if proxlb_config['vm_balancing_enable'] or proxlb_config['storage_balancing_enable'] or app_args.best_node:
node_statistics = get_node_statistics(api_object, proxlb_config['vm_ignore_nodes'])
vm_statistics = get_vm_statistics(api_object, proxlb_config['vm_ignore_vms'], proxlb_config['vm_balancing_type'])
node_statistics = update_node_statistics(node_statistics, vm_statistics)
# Calculate rebalancing of vms.
node_statistics_rebalanced, vm_statistics_rebalanced = balancing_calculations(balancing_method, balancing_mode, balancing_mode_option,
node_statistics, vm_statistics, balanciness, rebalance=False, processed_vms=[])
# Rebalance vms to new nodes within the cluster.
run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args, parallel_migrations)
# Execute VM balancing sub-routines.
if proxlb_config['vm_balancing_enable'] or app_args.best_node:
# Calculate rebalancing of vms.
node_statistics_rebalanced, vm_statistics_rebalanced = balancing_vm_calculations(proxlb_config['vm_balancing_method'], proxlb_config['vm_balancing_mode'], proxlb_config['vm_balancing_mode_option'],
node_statistics, vm_statistics, proxlb_config['vm_balanciness'], app_args, rebalance=False, processed_vms=[])
# Rebalance vms to new nodes within the cluster.
run_vm_rebalancing(api_object, vm_statistics_rebalanced, app_args, proxlb_config['vm_parallel_migrations'])
# Validate for any errors.
post_validations()
# Validate daemon service.
validate_daemon(daemon, schedule)
validate_daemon(proxlb_config['daemon'], proxlb_config['schedule'])
if __name__ == '__main__':

9
proxlb.conf
View File

@@ -4,11 +4,18 @@ api_user: root@pam
api_pass: FooBar
verify_ssl: 1
[balancing]
enable: 1
method: memory
mode: used
ignore_nodes: dummynode01,dummynode02
ignore_vms: testvm01,testvm02
[storage_balancing]
enable: 0
[update_service]
enable: 0
[api]
enable: 0
[service]
daemon: 1
schedule: 24
log_verbosity: CRITICAL
log_verbosity: CRITICAL