Kubectl exec in Cygwin gets "Upgrade request required"

We have several k8s clusters we run within our firewall. I’ve been using kubectl on Windows 10 in Cygwin for a long time to connect to our clusters. In general, this works perfectly fine. I’ve also been using “kubectl exec” for various purposes, also without any real trouble. I’m using v1.18.0 on the client, and v1.13.5 in the cluster.

Several days ago now, all of my attempts at “kubectl exec” are failing with “Upgrade request required”. I’ve noticed this error in multiple threads in the past, but I didn’t see any clear resolution for it. I did file an issue in the kubectl github for this.

Yesterday I tried running the resulting “kubectl exec” command line in a Windows cmd shell, and it works perfectly fine. Something about running the command in a Cygwin bash shell makes this happen, but note again that this only started happening a few days ago. I hadn’t updated anything in the client for a long time. I don’t believe the server-side has had a version change in a while, either.

I also talked to a colleague who has almost the same configuration and toolset, and he has not seen this problem.

I posted a question about this in the Cygwin list.

Note that I call kubectl in a bash scripting layer, I’m not typing the full “kubectl exec” command line.

The following is the last few lines of the log:

	I0614 09:12:44.355064   25788 round_trippers.go:423] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl.exe/v1.18.0 (windows/amd64) kubernetes/9e99141" -H "Authorization: Bearer ..." 'https://.../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
I0614 09:12:44.549412   25788 round_trippers.go:443] POST https://.../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true 400 Bad Request in 194 milliseconds
I0614 09:12:44.549412   25788 round_trippers.go:449] Response Headers:
I0614 09:12:44.549412   25788 round_trippers.go:452]     Content-Type: application/json
I0614 09:12:44.549412   25788 round_trippers.go:452]     Date: Sun, 14 Jun 2020 16:12:44 GMT
I0614 09:12:44.549412   25788 round_trippers.go:452]     Content-Length: 139
I0614 09:12:44.549412   25788 round_trippers.go:452]     Cache-Control: proxy-revalidate
I0614 09:12:44.549412   25788 round_trippers.go:452]     Proxy-Connection: Keep-Alive
I0614 09:12:44.549412   25788 round_trippers.go:452]     Connection: Keep-Alive
I0614 09:12:44.549412   25788 round_trippers.go:452]     Audit-Id: ecdb525f-20dd-4927-8848-a3c717381367
I0614 09:12:44.550405   25788 helpers.go:216] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Upgrade request required",
  "reason": "BadRequest",
  "code": 400
}]
F0614 09:12:44.550405   25788 helpers.go:115] Error from server (BadRequest): Upgrade request required

kubectl only guarantees compatibility with the last release. I’d downgrade to an older version.

Is he also running the same kubectl version?

Concerning the version compatibility issue, I had heard that, and I was actually following that rule for quite a while. I’m not sure why I diverged from it. However, it makes no difference in this case. I just installed v1.13.5 of kubectl and retested, and it made no difference.

My colleague is running v1.18.0.

I’ve managed to resolve this. I moved to version 1.14.0 of kubectl, and now it works. I assume that v1.13.5 would have agreed with the "within one minor version of the cluster version " rule. I also have no idea why it never failed in a Windows cmd shell.

That is really odd - from an api standpoint everything should have worked at that version – Would have to dig into what else was different from 1.13.5 to 1.14.0 to try and track it down =/

Unfortunately, it appears that this solution was only temporary. Now using v1.14.0, I’m getting “Upgrade request required” again, attempting to do “kubectl exec”.

You mentioned running inside a BASH script fails.
Does manually running directly in Cygwin also fail?

I’m not aware of any distinction there. I’m running in a Bash shell controlled by mintty, both provided by Cygwin.

And note that this isn’t an unconditional issue with the firewall between my laptop and our cluster, as I can run a Linux VM on my laptop and run the same script (albeit to a Linux install of kubectl, as opposed to the Windows version), and it works perfectly fine every time.

I was refering to your original problem statement:

Running from Windows CMD: success.
Running from BASH scripting layer (presumable in Cygwin): failure.
What about just opening Cygwin command prompt and running the kubectl exec command directly?

I don’t know how complex the BASH script is but maybe it’s setting some proxy or environment variables to cause the issue.
I’m only tying to narrow down the problem.

The error means that kubectl is not sending the correct HTTP request to upgrade the connection to a web socket, so presumably either kubectl is not sending the correct request, or else a proxy between your machine and the server is interfering with the HTTP request.

Since it works fine from Windows command prompt it suggests to me that your BASH script (or Cygwin) may be somehow directing calls through a proxy.

This is my conspiracy theory based on the info you’ve provided. :smile:

Kind regards,
Stephen

Sorry I wasn’t clear on this.

The scripting layer is just a convenience. If I run the raw “kubectl exec” command line, it fails in Cygwin, but not in the Windows cmd shell.

Also note that I can run a Linux VM on my laptop and run the same test (using the Linux kubectl, of course), and it also has no problem.

I tested with a server running 1.13.5 and client running various versions all within Cygwin and did not see similar symptoms.
The kubectl debug output for the request looks identical.

Most of the Google searches suggest that an LB is not forwarding the requests correctly to the API server.
I understand that it works from Windows command prompt which would still be going through the same LB, but can you just confirm where the cluster is running and if there is an LB in front of the API server?

When you run from Cygwin or command prompt they both using the exact same kubeconfig file? Both pointing to the exact same API server URL?

Now I’ve discovered that I’m now seeing the symptom from my Linux VM. This makes it clear to me there’s something about our firewall that is doing this, and that it has nothing to do with Cygwin. I’m now working with a platform engineer to see if he can repeat the same problem. The fact that this has “always” worked in the Windows CMD window could simply be because I haven’t tested it there very much.

I think I’ve managed to resolve this, with some hints from answers here and some other people. The problem is our proxy. If I simply turn it off, this works fine.

It’s very curious that it hasn’t always worked this way. We’ve recently been going through some changes in the proxy configuration, but that involved retiring an old proxy and setting to use a new one. I’ve been set to use the new one for quite a while now.

I don’t think it will be an issue to have the proxy unset in the shell, the only places I likely need the proxy is in my browser, and some other tools that would make external calls, but I think all of those have config files that would let me set the proxy for those connections.

Happy you got it sorted.
Thanks for updating with the details :+1: