Go Client - exec-ing a shel command in pod

General Discussions
1

Hello community,

I have the following code, leveraging the Kubernetes go client for executing a shell command in a pod programmatically:

// ExecuteRemoteCommand executes a remote shell command on the given pod
// returns the output from stdout and stderr
func ExecuteRemoteCommand(pod *v1.Pod, command string) (string, string, error) {
	kubeCfg := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
		clientcmd.NewDefaultClientConfigLoadingRules(),
		&clientcmd.ConfigOverrides{},
	)
	restCfg, err := kubeCfg.ClientConfig()
	if err != nil {
		return "", "", err
	}
	coreClient, err := corev1client.NewForConfig(restCfg)
	if err != nil {
		return "", "", err
	}

	buf := &bytes.Buffer{}
	errBuf := &bytes.Buffer{}
	request := coreClient.RESTClient().
		Post().
		Namespace(pod.Namespace).
		Resource("pods").
		Name(pod.Name).
		SubResource("exec").
		VersionedParams(&k8s_corev1.PodExecOptions{
			Command: []string{"/bin/sh", "-c", command},
			Stdin:   true,
			Stdout:  true,
			Stderr:  true,
			TTY:     true,
		}, scheme.ParameterCodec)
	exec, err := remotecommand.NewSPDYExecutor(restCfg, "POST", request.URL())
	err = exec.Stream(remotecommand.StreamOptions{
		Stdout: buf,
		Stderr: errBuf,
	})
	if err != nil {
		return "", "", errors.Wrapf(err, "Failed executing command %s on %v/%v", command, pod.Namespace, pod.Name)
	}

	return buf.String(), errBuf.String(), nil
}

I find there is some flakiness in this code - I sometimes get output from the shell command and I sometimes don’t. The command I’m using it ls, so it should always return some output.

exec.Stream() is synchronous, right?

Has anybody had any experience with similar functionality?

2

I’ve faced same question, the reason I found should be this: when you input the cmd to execute in container/pod, you may lost ‘LF’ or ‘CR’, cmd cannot be execute, so there is no output.

So I changed input methods, just like Kubernetes/dashboard, using sockJs to send cmd to pod/container. sockJs will do the job to add ‘LF’ or ‘CR’, every cmd input would be a whole executable cmd, so we can get the correct output.

3

I realized what the problem in that piece of code was - it’s the stdin argument. Setting it to false fixed my issue.

4

For me it seems like this snippet fails indeed on the SPDY negotiation part. Even with STDIN=true;

Any other ideas for a robust kubectl exec in golang?

When i looked into the code, this seems to ultimately return a ‘nil’ value

func  (s *SpdyRoundTripper) NewConnection(resp *http.Response) (httpstream.Connection, error) {
	connectionHeader := strings.ToLower(resp.Header.Get(httpstream.HeaderConnection))
	upgradeHeader := strings.ToLower(resp.Header.Get(httpstream.HeaderUpgrade))
	if (resp.StatusCode != http.StatusSwitchingProtocols) || !strings.Contains(connectionHeader, strings.ToLower(httpstream.HeaderUpgrade)) || !strings.Contains(upgradeHeader, strings.ToLower(HeaderSpdy31)) {
		defer resp.Body.Close()
		responseError := ""
		responseErrorBytes, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			fmt.Println("whoops ")
			responseError = "unable to read error from server response"
			fmt.Println(err, responseError)
		} else {
			// TODO: I don't belong here, I should be abstracted from this class

			if obj, _, err := statusCodecs.UniversalDecoder().Decode(responseErrorBytes, nil, &metav1.Status{}); err == nil {
				if status, ok := obj.(*metav1.Status); ok {
					fmt.Println("why are we returning nil????", *status, ok, obj)
					return nil, &apierrors.StatusError{ErrStatus: *status}
				}

I suppose that returning a nil value isnt ‘good’ but is somehow valid, and then something breaks downstream from this. Note im running client-go on ubuntu 18, and talking to a Kind cluster. Not sure if that somehow might result in
SPDY behaving badly.

Update

So, after digging more , i saw that the NewConnection function in rounttrippers.go, actually returns

func (s *SpdyRoundTripper) NewConnection(resp *http.Response) (httpstream.Connection, error) {

The following:

enter  &{404 Not Found 404 HTTP/1.1 1 1 map[Content-Length:[487] Content-Type:[application/json] Date:[Mon, 17 Feb 2020 04:06:04 GMT]] 0xc000552200 487 [] false false map[] <nil> <nil>}

Hence somehow the upgrade request is getting 404’d.

5

For what it’s worth, I had to modify Stanislav’s code to get it to work. I think there’s been some updates to the api since his answer or something. Here’s a version that works for me.

import (
	"bytes"
	"fmt"

	v1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/kubernetes/scheme"
	"k8s.io/client-go/tools/clientcmd"
	"k8s.io/client-go/tools/remotecommand"
)

func ExecuteRemoteCommand(pod *v1.Pod, command string) (string, string, error) {
	kubeCfg := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
		clientcmd.NewDefaultClientConfigLoadingRules(),
		&clientcmd.ConfigOverrides{},
	)
	restCfg, err := kubeCfg.ClientConfig()
	if err != nil {
		return "", "", err
	}
	coreClient, err := kubernetes.NewForConfig(restCfg)
	if err != nil {
		return "", "", err
	}

	buf := &bytes.Buffer{}
	errBuf := &bytes.Buffer{}
	request := coreClient.CoreV1().RESTClient().
		Post().
		Namespace(pod.Namespace).
		Resource("pods").
		Name(pod.Name).
		SubResource("exec").
		VersionedParams(&v1.PodExecOptions{
			Command: []string{"/bin/sh", "-c", command},
			Stdin:   false,
			Stdout:  true,
			Stderr:  true,
			TTY:     true,
		}, scheme.ParameterCodec)
	exec, err := remotecommand.NewSPDYExecutor(restCfg, "POST", request.URL())
	err = exec.Stream(remotecommand.StreamOptions{
		Stdout: buf,
		Stderr: errBuf,
	})
	if err != nil {
		return "", "", fmt.Errorf("%w Failed executing command %s on %v/%v", err, command, pod.Namespace, pod.Name)
	}

	return buf.String(), errBuf.String(), nil
}
1 Like
6

@MatthewTromp thank you for your example. That worked for me as well!