Issues with kubectl (v1.12) exec: unable to upgrade connection: Forbidden user

General Discussions
1

Hi,

I’ve been learning about Kubernetes using Kelsey Hightower’s excellent kubernetes-the-hard-way-guide.

Using this guide I’ve installed v1.12 on GCE. Everything works perfectly apart from kubectl exec:

kubectl exec -it shell-demo – /bin/bash --kubeconfig=/root/certsconfigs/admin.kubeconfig
error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)

Note that I have KUBECONFIG=/root/certsconfigs/admin.kubeconfig

Apart from exec all other kubectl functions work as expected with this admin.kubeconfig file, so from that I deduce it valid for use with my cluster.

I’m pretty sure I have made a beginners mistake somehere, but if somebody could advise where I have gone awry I should be most grateful.

TIA

Shaun

I have double checked no .kube/config file exists anywhere on my Master Controller.

root@controller-1:/root/deployment/kubernetes# kubectl get pods
NAME READY STATUS
shell-demo 1/1 Running 0 23m

Here is the output with -v8
root@controller-1:/root/deployment/kubernetes# kubectl -v8 exec -it shell-demo – /bin/bash
I1118 15:18:16.898428 11117 loader.go:359] Config loaded from file /root/certsconfigs/admin.kubeconfig
I1118 15:18:16.899531 11117 loader.go:359] Config loaded from file /root/certsconfigs/admin.kubeconfig
I1118 15:18:16.900611 11117 loader.go:359] Config loaded from file /root/certsconfigs/admin.kubeconfig
I1118 15:18:16.902851 11117 round_trippers.go:383] GET ://127.0.0.1:6443/api/v1/namespaces/default/pods/shell-demo
I1118 15:18:16.902946 11117 round_trippers.go:390] Request Headers:
I1118 15:18:16.903016 11117 round_trippers.go:393] Accept: application/json, /
I1118 15:18:16.903091 11117 round_trippers.go:393] User-Agent: kubectl/v1.12.0 (linux/amd64) kubernetes/0ed3388
I1118 15:18:16.918699 11117 round_trippers.go:408] Response Status: 200 OK in 15 milliseconds
I1118 15:18:16.918833 11117 round_trippers.go:411] Response Headers:
I1118 15:18:16.918905 11117 round_trippers.go:414] Content-Type: application/json
I1118 15:18:16.918974 11117 round_trippers.go:414] Content-Length: 2176
I1118 15:18:16.919053 11117 round_trippers.go:414] Date: Sun, 18 Nov 2018 15:18:16 GMT
I1118 15:18:16.919218 11117 request.go:942] Response Body: {“kind”:“Pod”,“apiVersion”:“v1”,“metadata”:{“name”:“shell-demo”,“namespace”:“default”,“selfLink”:"/api/v1/namespaces/default/pods/shell-demo",“uid”:“99f320f8-eb42-11e8-a053-42010af0000b”,“resourceVersion”:“13213”,“creationTimestamp”:“2018-11-18T14:59:51Z”},“spec”:{“volumes”:[{“name”:“shared-data”,“emptyDir”:{}},{“name”:“default-token-djprb”,“secret”:{“secretName”:“default-token-djprb”,“defaultMode”:420}}],“containers”:[{“name”:“nginx”,“image”:“nginx”,“resources”:{},“volumeMounts”:[{“name”:“shared-data”,“mountPath”:"/usr/share/nginx/html"},{“name”:“default-token-djprb”,“readOnly”:true,“mountPath”:"/var/run/secrets/kubernetes.io/serviceaccount"}],“terminationMessagePath”:"/dev/termination-log",“terminationMessagePolicy”:“File”,“imagePullPolicy”:“Always”}],“restartPolicy”:“Always”,“terminationGracePeriodSeconds”:30,“dnsPolicy”:“ClusterFirst”,“serviceAccountName”:“default”,“serviceAccount”:“default”,“nodeName”:“worker-1”,“securityContext”:{},“schedulerName”:“default-scheduler”,“tolerations”:[{“key”:"node.kubernet [truncated 1152 chars]
I1118 15:18:16.925240 11117 round_trippers.go:383] POST …

error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)

1 Like
2

I had the same problem today and though you have probably moved on, I’ll share what I did.

found out what node was running the deployment with
kubectl describe deployment shell

I ssh’ed on the node and confirmed that the certificates for the kubelet were there and valid with something like
openssl x509 -text -noout -in worker-1.pem

I looked at the journal for the kubelet service for any errors with something like
journalctl -n 10000 -u kubelet -xel

I didn’t see anything out of the ordinary, so I thought maybe it had to do with RBAC. So I went back to a controller node and ran
kubectl get clusterrole

THERE, I saw that the clusterroles were not defined. I totally missed setting up the clusterrole binding in the bootstrapping controller node step in “the hard way.”

So, that’s my experience/troubleshooting as I’m learning this very cool system. Hope that’s helpful to someone.

1 Like
3

The error you’re encountering with kubectl exec - “unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)” - typically indicates an issue with Role-Based Access Control (RBAC) permissions in your Kubernetes setup. It seems that the kubernetes user does not have the necessary permissions to perform the requested action.

Given that you are following Kelsey Hightower’s “Kubernetes The Hard Way,” and other kubectl functions are working correctly with your admin.kubeconfig, the problem likely lies in the specific RBAC settings related to the exec command. Here’s how you can troubleshoot and potentially solve this issue:

1. Check RBAC Settings for the User

  • The kubernetes user needs specific permissions to create a proxy to nodes, which is required for the kubectl exec command to work.
  • Ensure that the user has a role with the necessary permissions (like create proxy on nodes) and that the role is bound to the user correctly.

2. Review the Role and RoleBinding

  • Examine the Role (or ClusterRole) and RoleBinding (or ClusterRoleBinding) associated with the kubernetes user. You can use the following commands:
    • kubectl get clusterroles and kubectl get clusterrolebindings to list roles and role bindings at the cluster level.
    • kubectl describe clusterrole [role_name] to view the permissions of a specific role.
    • kubectl describe clusterrolebinding [rolebinding_name] to see which users or groups are bound to a role.

3. Update RBAC Configuration if Necessary

  • If the kubernetes user does not have the required permissions, you’ll need to update the RBAC configuration.
  • You can either modify an existing role to include the necessary permissions or create a new role and role binding for the kubernetes user.

4. Verify the Node Proxy Subresource

  • The error message specifically mentions resource=nodes, subresource=proxy. This implies that the user needs permission to access the proxy subresource of nodes.
  • Ensure that the role includes permissions like "get", "list", "watch", "create", "update", "patch", "delete" on nodes/proxy.