Simple sidecar application

[horos22]

All,

I had a very simple sidecar application which - frustratingly - doesn’t look so simple to implement in kubernetes. Inside a kubernetes cluster I want to:

1. do the equivalent of a 'kubectl get pods' for the current namespace the pod is on
2. check the liveness of each container inside the pod
3. run a series of commands inside each of the individual containers in the pod, depending on the container doing a different set of monitoring commands
4.  send the results to either sumo logic or to a prometheus endpoint.

Again, I want to do all of these things INSIDE A STANDALONE CONTAINER, which seems to be the rub here. You can do kubectl commands but only on resources outside the pod itself. Which means I need to worry about authentication, kube files, etc. If I’m inside the pod itself I shouldn’t need to worry about such things which simplifies setup a lot.

This is annoyingly difficult. There doesn’t seem to be a standard way for containers to communicate this way over the common network for the namespace, nor again does there seem to be a way to actually DO the equivalent ‘kubectl get pods’ inside a container. AFAICT, I’d have to setup hacks for both of these things and I don’t want to do this.

Is this functionality like this in newer versions of kubernetes? Or is there a prebuilt container which does something like this? Or am I just missing something obvious?

Thanks much - and very confused -

Greg

[thockin]

I am a little confused. You say “sidecar” but then you also say “get pods”. Is this a container that you run alongside “real” workloads to monitor them? Or is this something that runs on its own and pokes at other pods?

You can use the kube API from within a pod - you can get Service account credentials mounted and use any client lib or kubectl literally.

If the goal is to have a shortcut for “tell me about THIS pod”, we have the “downward API” but it does not have everything exposed. There is not a shortcut to “exec in a peer container”, for example.

[horos22]

Thockin -

Thanks for responding - Here is my use case in python pseudo code. Its a simple loop, getting the containers in the pod’s namespace, testing the status of those pods, and exec’ing commands on those individual pods to generate a report for prometheus. So it is a container that runs on its own and pokes at other containers.

AFAICT, ‘kubectl get pods -n ’ will give you a list of containers running in the namespace , whether they are ready or not, their status, how many restarts they have had and their age.

So any pointers here to docs that I can look at would be greatly appreciated. I see the ‘downward api’, but offhand, it looks very intrusive. A good sidecar is one that doesn’t have to modify the whole car, and what I want is a container that I can drop in place - standalone - in any environment without needing to change the pods configuration that hosts it. Or if it does, changes it only in a way which is minimal.

Plus downward api doesn’t really go into a method of actually running calls on remote containers. ssh is painful because of key setup, and I don’t want to host my own connection method. There should be a lightweight, trusted and secure method that doesn’t require any validation and that runs transparently inside the network setup via kubernetes. After all it is a pod - the containers internal to the pod should be able to work in sync with each other.

I don’t know if this exists, but if it doesn’t it should. Here’s the pseudocode:

while True:

os.sleep(300);
for _instance in get_containers_in_namespace():

    status = check_status_of_container(_instance).  # check if ready, down, crash-loop, etc
    
    if status == 'ready':
         metrics = run_metrics_job_on_container(_instance)
         post_metrics_to_prometheus(metrics)
    else:
         post_error_status_to_prometheus(status)

[thockin]

| horos22
July 19 |

• | - |

Thockin -

Thanks for responding - Here is my use case in python pseudo code. Its a simple loop, getting the containers in the pod’s namespace, testing the status of those pods, and exec’ing commands on those individual pods to generate a report for prometheus. So it is a container that runs on its own and pokes at other containers.

Cool, so it’s not a “sidecar” it’s just a kube API client.

AFAICT, ‘kubectl get pods -n ’ will give you a list of containers running in the namespace , whether they are ready or not, their status, how many restarts they have had and their age.

Correct - is that not what you are asking for? You can read their latest statuses, what images they are running, etc.

So any pointers here to docs that I can look at would be greatly appreciated.

Have you looked at the many client-go examples? Assuming you want to code in Go, that is.

I see the ‘downward api’, but offhand, it looks very intrusive. A good sidecar is one that doesn’t have to modify the whole car, and what I want is a container that I can drop in place - standalone - in any environment without needing to change the pods configuration that hosts it. Or if it does, changes it only in a way which is minimal.

You switched back to talking about sidecars, but above you said it was it’s own thing?

Plus downward api doesn’t really go into a method of actually running calls on remote containers. ssh is painful because of key setup, and I don’t want to host my own connection method. There should be a lightweight, trusted and secure method that doesn’t require any validation and that runs transparently inside the network setup via kubernetes. After all it is a pod - the containers internal to the pod should be able to work in sync with each other.

So, wait, it IS a sidecar? Are you poking at other containers exclusively in THIS pod? Or poking at other pods in this (or another) namespace?

Downward API is a way to learn information about your own pod - name, namespace, etc. It does not have “active” calls like exec.

While, yes, the pod real a designed to work together, it’s not assumed that code running in one container can or should be able to access peer containers via the kube API.

You need to grant that pod’s ServiceAccount access to do the things you want via RBAC (usually) and then transit the kube API.

AFAIK there is no shortcut.

I don’t know if this exists, but if it doesn’t it should. Here’s the pseudocode:

while True:

os.sleep(300);

for _instance in get_containers_in_namespace():

This is the confusion - containers are not in a namespace. They are in pods, and pods are in the namespace . It’s a huge difference to say “get containers in this one pod” vs “get containers in all pods in this namespace”

[horos22]

Correct - is that not what you are asking for? You can read their latest statuses, what images they are running, etc.

So any pointers here to docs that I can look at would be greatly appreciated.

Have you looked at the many client-go examples? Assuming you want to code in Go, that is.

I think there might be a slight difference in how we define terms here. To me, a sidecar provides standalone and useful functionality such that you don’t have to build in that functionality into each and every client the side car interacts with. In that sense, my application is a very straightforward sidecar. I do the following:

1. query the pod that I’m in for all sibling containers which I attach to.
2. provide functionality for all of these sibling containers by exec-ing code on them and gleaning results.

If I need to program a go client/server in order to communicate between my sidecar and its clients, then I multiply my provisioning requirements proportional to the number of clients that my sidecar is providing services for. If I have 100 types of clients, I have to provide them with 100 go servers which handle requests, and I need to provision my go program locally in my sidecar. ie: I need to modify 100 Dockerfiles. Plus I need to either find one or write one, when I really want something standard.

Having to do all this adhoc work - instead of kubernetes providing a default solution akin to what it provides with kubectl - makes my sidecar a lot more clunky. It now has to impose a contract on all of these other systems, one that really shouldn’t need to be there since you should (imo) be able to run kubectl internally on any container if configured to do so. And that would include exec, get pods, etc.

You switched back to talking about sidecars, but above you said it was it’s own thing?

I said that it is a sidecar in the sense that it is modifying its clients and providing functionality. And just like any good generics implementation, the less assumptions it makes about its client the better. I want a generic sidecar, one that has the least number of hoops that need to be jumped through in order to work. The more hoops it needs to go through, the less chance that it will have in taking off.

While, yes, the pod real a designed to work together, it’s not assumed that code running in one container can or should be able to access peer containers via the kube API.

Yes, and that is in my mind a major lost opportunity. Its easy enough, all you would have to do is have an internal directive on the pod that says:

SideCarPod: True

which handles all the plumbing for you, uses a standard method for connecting between the pods (possibly kubectl exec), avoids the need to touch an api at all, makes it so you don’t need to have RBAC, etc. You could still do what you are saying is necessary right now, but you wouldn’t need to. In fact I don’t see why you would even WANT to given how much easier it would be to provide a single directive at the appropriate point here.

This is the confusion - containers are not in a namespace. They are in pods, and pods are in the
namespace . It’s a huge difference to say “get containers in this one pod” vs “get containers in all pods
in this namespace”

fair enough, I was sloppy with my terminology. However I’m hoping it is clear here what I’m asking for now.

I really want to hear what the devs’ opinions are on this. AFAICT it seems an obvious productivity win. The one thing I could see as an argument is that it skimps on security, but:

 a. you wouldn't need to use it
 b.if an attacker has gotten a shell to a given pod that doesn't have an ssh connection to the outside world, they most likely have the kube config and your day is already ruined.
 c. by providing your own connection method for communication you are introducing a security vulnerability of your own if it is based on a listening port.

Frankly it seems that communication between containers in a pod is a fairly basic requirement here, and not having a fairly straightfoward solution here out of the box and forcing end users to implement their own for even basic applications is puzzling. Don’t get me wrong, I love kubernetes but having this would make a lot of people’s lives easier IMO.

[thockin]

Sorry for any formatting issues - I am on mobile

| horos22
July 20 |

• | - |

Correct - is that not what you are asking for? You can read their latest statuses, what images they are running, etc.

So any pointers here to docs that I can look at would be greatly appreciated.

Have you looked at the many client-go examples? Assuming you want to code in Go, that is.

I think there might be a slight difference in how we define terms here. To me, a sidecar provides standalone and useful functionality such that you don’t have to build in that functionality into each and every client the side car interacts with. In that sense, my application is a very straightforward sidecar. I do the following:

1. query the pod that I’m in for all sibling containers which I attach to.

Cool, this statement clarified a lot for me. It IS a sidecar, and it is interacting exclusively with “peer” container(s) in the same pod. Thanks.

1. provide functionality for all of these sibling containers by exec-ing code on them and gleaning results.

So, no matter what solution we cook up, you need to make some API call which runs a command in the target container, and which streams input into that executing command and output/errors from it. Right? Then you have to parse the output and some something with it.

If I need to program a go client/server in order to communicate between my sidecar and its clients, then I multiply my provisioning requirements proportional to the number of clients that my sidecar is providing services for. If I have 100 types of clients, I have to provide them with 100 go servers which handle requests, and I need to provision my go program locally in my sidecar. ie: I need to modify 100 Dockerfiles. Plus I need to either find one or write one, when I really want something standard.

Some confusion. I meant that your sidecar can use the Kubernetes client library in Go (or some other language, but I know less about those client libs) to talk to the Kubernetes API server to run the exec into your peer container.

I know you don’t like the idea of going up and out to the API server just I come back down to the same pod, but I think that’s the best option that works today. I don’t know if a shortcut.

Having to do all this adhoc work - instead of kubernetes providing a default solution akin to what it provides with kubectl -

If you don’t like writing your own kube API client, you can literally run kubectl. All you need to do is write an RBAC Role and RoleBinding which grants the ServiceAccount access to pods/exec.

I will admit I am not and expert in all the features of the RBAC system, so I am not sure if there is a way to express “only to my same pod” or “only to pods of this same SA”. I suspect not, which means you would be granting access to all pods in the same NS, as a first cut.

While, yes, the pod real a designed to work together, it’s not assumed that code running in one container can or should be able to access peer containers via the kube API.

Yes, and that is in my mind a major lost opportunity. Its easy enough, all you would have to do is have an internal directive on the pod that says:

SideCarPod: True

which handles all the plumbing for you, uses a standard method for connecting between the pods (possibly kubectl exec), avoids the need to touch an api at all, makes it so you don’t need to have RBAC, etc. You could still do what you are saying is necessary right now, but you wouldn’t need to. In fact I don’t see why you would even WANT to given how much easier it would be to provide a single directive at the appropriate point here.

KEPs are welcome. It’s not a terrible idea, but the details are more complicated than you are assuming, I think. I have not seen many instances of people who need to exec into a peer container, but it’s not terribly implausible.

More often I see either exec probes, which kubelet already knows how to do, or I see sidecars that poke at peers over localhost network.

E.g. main app binds to localhost:port and exposes native stats, and the sidecar concerts those to the preferred export format (e.g. Prometheus).

Another path is to expose a UNIX socket on an emptyDir volume and have a sidecar use that.

I really want to hear what the devs’ opinions are on this. AFAICT it seems an obvious productivity win. The one thing I could see as an argument is that it skimps on security, but:

 a. you wouldn't need to use it

 b.if an attacker has gotten a shell to a given pod that doesn't have an ssh connection to the outside world, they most likely have the kube config and your day is already ruined.

 c. by providing your own connection method for communication you are introducing a security vulnerability of your own if it is based on a listening port.

A is the saving grace for the idea.

B is not true - RCEs happen all the time.

Frankly it seems that communication between containers in a pod is a fairly basic requirement here, and not having a fairly straightfoward solution here out of the box and forcing end users to implement their own for even basic applications is puzzling. Don’t get me wrong, I love kubernetes but having this would make a lot of people’s lives easier IMO.

As I mentioned, we have lots of mechanisms for communication within a pod: network, volumes, signals, IPC, etc. Just not exec.

You might argue it is conspicuous in its absence, but I think it’s an order of magnitude more complex and and order of magnitude (or 2 or 3) less commonly needed.

I encourage you to start a KEP - work out the details of how it would actually work as securely as possible.

Tim

[B_Ram]

apiVersion: v1
kind: Pod
metadata:
name: sidecar-container-demo
spec:
containers:

• image: busybox
  command: [“/bin/sh”]
  args: [“-c”, “while true; do echo echo $(date -u) ‘Hi I am from Sidecar container’ >> /var/log/index.html; sleep 5;done”]
  name: sidecar-container
  resources: {}
  volumeMounts:
  • name: var-logs
    mountPath: /var/log
• image: nginx
  name: main-container
  resources: {}
  ports:
  • containerPort: 80
    volumeMounts:
  • name: var-logs
    mountPath: /usr/share/nginx/html
    dnsPolicy: Default
    volumes:
• name: var-logs
  emptyDir: {}