Intra cluster service migration


Let me explain my scenario. I have two Kubernetes clusters 1&2 - one master and two worker nodes running on each cluster. I want to move the services from one cluster to another cluster. For ex, now the service requests are processed in cluster 1(inside the pod of worker node number 2). Capture the running containers as an image and move it to cluster 2. Create a pod in the worker node of cluster 2 and make the services resume and run. Once the requests are processed, the response message should be sent to the user from cluster 2. The user should not know about the service movement that happened.

My question is what the step are the steps to follow to move the services and establish internal communication between the clusters. There is not many references about the cluster communication. I used docker commit command to snapshot an image and stuck with the further steps. Kindly help


My environment setup consists OS/Kernel Linux Ubuntu 18.04, Tegra 4.9 Kubernetes v1.16 Docker 18.09.7 .I use Jetson Xavier for all master and worker nodes(8-core ARMv8.2 8 MB L2+4 MB L3, 16 GB LPDDR4, GPU - 512-core Volta with Tensor Core & Network module Intel AC9560, AGW 200)

You said a few things here.

I want to move the services from one cluster to another cluster

This is very reasonable, but Kubernetes does not have out-of-the-box support for this (because it depends a LOT on your network setup).

Capture the running containers as an image and move it to cluster 2.

This is a scary statement - it assumes you are treating pods as pets with important mutable state in the image. I’d strongly suggest to not do that.

What do you use to bring traffic into the cluster? Are your clusters’ pod networks flat or are you using overlays? For example, on Google Cloud we call this multi-cluster ingress. We configure a load-balancer to route to available backends in both clusters. Then you can draw down the number of backends in the old cluster.

There’s a concept of multi-cluster service, which is still pretty early, which you could maybe use for in-cluster ingress controllers, but I don’t think anyone is explicitly supporting that yet.

Thank you so much for your response.
My goal is to move the services from one cluster to another.
I have one cluster set up now where I use the load balancer to manage the traffic.
Moving through a snapshot was my idea. As it is not recommended, I drop that idea.
Could you please explain to me more on how can the experiment be performed?

You did not explain what sort of network you use for pods or for LB. The “simple” answer is to have the LB target pods in both clusters, but I don’t know enough to say if that will work for you or not.