Are there any (free/FOSS/CNCF-sponsored) tools that would allow two separate clusters to be joined in order to better split the workload?
In summary: I have an on-premise K8S cluster (v1.16) with a few hundred nodes that are used for CPU-bound tasks. There are cases where, in order to meet certain deadlines, we’d like to dynamically spin-up an AWS-based cluster on-demand, and “join” it to a remote cluster. We are currently using WeaveNet, and not much else. Is there a way, using free/FOSS/CNCF-sponsored tools/plugins/etc., to “join”/bridge our on-prem cluster (on-demand, without having to re-create the clusters) with an arbitrary remote custom/AWS/Azure-based cluster, such that we can later disconnect the two clusters without having to re-create/provision them?
The on-premise cluster is not able to allow incoming firewall rules/exceptions, but the remote cluster(s) is/are allowed to dynamically reconfigure firewall/routing settings to allow the “gold” cluster to initiate a temporary (i.e. 2-4 hours in duration) join/bridge operation so that processing tasks can be distributed among the nodes in the joined/mixed cluster.