The connection to the server <host>:6443 was refused - did you specify the right host or port?

hi,maybe you can check this topic

1 Like

Hi,
Thanks for the replay and suggested topic clears my doubt.

Hi, I I’m facing the same problem and I’m pretty sure @changec’s way does not work (there is no swap partition
It there another solution?

Hi, I have the same problem and listed above solution did not help me. My system runs on VMware Workstation 15 Player:

denis@ubuntu:~/dev$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic

When I try to execute kubectl command:

denis@ubuntu:~/dev$ kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?

The logs of restarted kubelet:

янв 30 15:58:45 ubuntu systemd[1]: Started kubelet: The Kubernetes Node Agent.
янв 30 15:58:45 ubuntu kubelet[7651]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
янв 30 15:58:45 ubuntu kubelet[7651]: Flag --allow-privileged has been deprecated, will be removed in a future version
янв 30 15:58:45 ubuntu kubelet[7651]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
янв 30 15:58:45 ubuntu kubelet[7651]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
янв 30 15:58:45 ubuntu kubelet[7651]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
янв 30 15:58:45 ubuntu kubelet[7651]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
янв 30 15:58:45 ubuntu kubelet[7651]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.12, and the cadvisor port will be removed entirely in 1.13
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.666637    7651 feature_gate.go:226] feature gates: &{{} map[]}
янв 30 15:58:45 ubuntu kubelet[7651]: W0130 15:58:45.676051    7651 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
янв 30 15:58:45 ubuntu kubelet[7651]: W0130 15:58:45.681294    7651 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.681495    7651 server.go:376] Version: v1.10.5
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.681615    7651 feature_gate.go:226] feature gates: &{{} map[]}
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.681772    7651 plugins.go:89] No cloud provider specified.
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.683749    7651 certificate_store.go:117] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.731337    7651 server.go:614] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.731677    7651 container_manager_linux.go:242] container manager verified user specified cgroup-root exists: /
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.731697    7651 container_manager_linux.go:247] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true}
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.731866    7651 container_manager_linux.go:266] Creating device plugin manager: true
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.731907    7651 state_mem.go:36] [cpumanager] initializing new in-memory state store
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.731963    7651 state_mem.go:84] [cpumanager] updated default cpuset: ""
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.731990    7651 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.732149    7651 kubelet.go:273] Adding pod path: /etc/kubernetes/manifests
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.732175    7651 kubelet.go:298] Watching apiserver
янв 30 15:58:45 ubuntu kubelet[7651]: E0130 15:58:45.739359    7651 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Get https://192.168.10.68:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.10.68:6443: getsockopt: connection refused
янв 30 15:58:45 ubuntu kubelet[7651]: E0130 15:58:45.739438    7651 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.10.68:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dubuntu&limit=500&resourceVersion=0: dial tcp 192.168.10.68:6443: getsockopt: connection refused
янв 30 15:58:45 ubuntu kubelet[7651]: E0130 15:58:45.739668    7651 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Failed to list *v1.Node: Get https://192.168.10.68:6443/api/v1/nodes?fieldSelector=metadata.name%3Dubuntu&limit=500&resourceVersion=0: dial tcp 192.168.10.68:6443: getsockopt: connection refused
янв 30 15:58:45 ubuntu kubelet[7651]: W0130 15:58:45.740027    7651 kubelet_network.go:139] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.740140    7651 kubelet.go:558] Hairpin mode set to "hairpin-veth"
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.741209    7651 client.go:75] Connecting to docker on unix:///var/run/docker.sock
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.741295    7651 client.go:104] Start docker client with request timeout=2m0s
янв 30 15:58:45 ubuntu kubelet[7651]: W0130 15:58:45.742361    7651 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
янв 30 15:58:45 ubuntu kubelet[7651]: W0130 15:58:45.744173    7651 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.745059    7651 docker_service.go:244] Docker cri networking managed by kubernetes.io/no-op
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.756618    7651 docker_service.go:249] Docker Info: &{ID:PCAF:JPRA:TPT2:IMHM:QVJV:F4HB:MAR4:7LQ4:556Z:I4SC:3CHJ:CR5G Containers:49 ContainersRunning:0 ContainersPaused:0 ContainersStopped:49 Images:43 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog]} MemoryLimit:true SwapLimit:false KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:21 OomKillDisable:true NGoroutines:34 SystemTime:2019-01-30T15:58:45.751810456+03:00 LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.15.0-44-generic OperatingSystem:Ubuntu 18.04.1 LTS OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc42055f2d0 NCPU:2 MemTotal:4112048128 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu Labels:[] ExperimentalBuild:false ServerVersion:18.03.1-ce ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:docker-runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:773c489c9c1b21a6d78b5c538cd395416ec50f88 Expected:773c489c9c1b21a6d78b5c538cd395416ec50f88} RuncCommit:{ID:4fc53a81fb7c994640722ac585fa9ca548971871 Expected:4fc53a81fb7c994640722ac585fa9ca548971871} InitCommit:{ID:949e6fa Expected:949e6fa} SecurityOptions:[name=apparmor name=seccomp,profile=default]}
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.757187    7651 docker_service.go:262] Setting cgroupDriver to cgroupfs
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.780788    7651 remote_runtime.go:43] Connecting to runtime service unix:///var/run/dockershim.sock
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.781780    7651 kuberuntime_manager.go:186] Container runtime docker initialized, version: 18.03.1-ce, apiVersion: 1.37.0
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.782118    7651 csi_plugin.go:63] kubernetes.io/csi: plugin initializing...
янв 30 15:58:45 ubuntu kubelet[7651]: E0130 15:58:45.785024    7651 kubelet.go:1282] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.785473    7651 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.785497    7651 status_manager.go:140] Starting to sync pod status with apiserver
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.785505    7651 kubelet.go:1782] Starting kubelet main sync loop.
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.785513    7651 kubelet.go:1799] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.785751    7651 server.go:129] Starting to listen on 0.0.0.0:10250
янв 30 15:58:45 ubuntu kubelet[7651]: E0130 15:58:45.785898    7651 event.go:209] Unable to write event: 'Post https://192.168.10.68:6443/api/v1/namespaces/default/events: dial tcp 192.168.10.68:6443: getsockopt: connection refused' (may retry after sleeping)
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.786202    7651 server.go:299] Adding debug handlers to kubelet server.
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.787063    7651 volume_manager.go:247] Starting Kubelet Volume Manager
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.787905    7651 server.go:945] Started kubelet
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.788048    7651 desired_state_of_world_populator.go:129] Desired state populator starts to run
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.886497    7651 kubelet.go:1799] skipping pod synchronization - [container runtime is down]
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.887527    7651 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
янв 30 15:58:45 ubuntu kubelet[7651]: I0130 15:58:45.889846    7651 kubelet_node_status.go:82] Attempting to register node ubuntu
янв 30 15:58:45 ubuntu kubelet[7651]: E0130 15:58:45.890164    7651 kubelet_node_status.go:106] Unable to register node "ubuntu" with API server: Post https://192.168.10.68:6443/api/v1/nodes: dial tcp 192.168.10.68:6443: getsockopt: connection refused

When I restart kubelet I observe that “k8s” containers start but after several seconds automatically stop.

denis@ubuntu:~/dev$ docker container ls
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS               NAMES
1d0275629b45        4fb6852fef47                      "kube-apiserver --re…"   18 seconds ago      Up 18 seconds                           k8s_kube-apiserver_kube-apiserver-ubuntu_kube-system_54717c7acb06777aaa0da90e6b43d518_1
1a19e90b1ff0        k8s.gcr.io/etcd-amd64             "etcd --advertise-cl…"   21 seconds ago      Up 20 seconds                           k8s_etcd_etcd-ubuntu_kube-system_e7aaf2590e8bec8679770ef31422f566_0
7b3c17234b54        k8s.gcr.io/kube-scheduler-amd64   "kube-scheduler --ad…"   37 seconds ago      Up 36 seconds                           k8s_kube-scheduler_kube-scheduler-ubuntu_kube-system_ab7798e80dac8c9d88788f8e132924b1_0
805ee8bf5e38        k8s.gcr.io/pause-amd64:3.1        "/pause"                 41 seconds ago      Up 39 seconds                           k8s_POD_kube-apiserver-ubuntu_kube-system_54717c7acb06777aaa0da90e6b43d518_0
a6daeb3ba870        k8s.gcr.io/pause-amd64:3.1        "/pause"                 41 seconds ago      Up 39 seconds                           k8s_POD_etcd-ubuntu_kube-system_e7aaf2590e8bec8679770ef31422f566_0
0762c7cfc6f6        k8s.gcr.io/pause-amd64:3.1        "/pause"                 41 seconds ago      Up 39 seconds                           k8s_POD_kube-scheduler-ubuntu_kube-system_ab7798e80dac8c9d88788f8e132924b1_0

The kubeadm version is:

denis@ubuntu:~/dev$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:34:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
denis@ubuntu:~/dev$ 

The kubectl version:

denis@ubuntu:~/dev$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

The kubelet version:

denis@ubuntu:~/dev$ kubelet version
I0130 17:45:00.046730   12766 feature_gate.go:226] feature gates: &{{} map[]}
F0130 17:45:00.046938   12766 server.go:218] error reading /var/lib/kubelet/pki/kubelet.key, certificate and key must be supplied as a pair

The docker version is:

denis@ubuntu:~/dev$ docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Wed Jun 20 21:43:51 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Wed Jun 20 21:42:00 2018
  OS/Arch:      linux/amd64
  Experimental: false
denis@ubuntu:~/dev$ 

Please, help. I’m ready to provide any needed information.

Hello, after increasing disc space the problem was solved. Unfortunately, in the logs I did not see the records related to the problem of disk space. Five days were spent on finding out the cause, since it is not obvious.

2 Likes
kubectl get nodes

may have shown the node has diskpressure. Something i didn’t see in your post.

azzaka, thank you for answer, but “cube get nodes” was useless for my case, cause cluster did not start properly (all main modules as apiserver, etcd etc was not run). So I was getting “connection refused,” as mentioned in the first post.

I have the hit the same issue on CentOS 7 even after disabling the swap.

But i managed to run kubectl get nodes by explicitly passing --kubeconfig parameter. This is not as per the kubernetes docs

I have set KUBECONFIG file but kubectl does care about that parameter.

I had the exact same error today…and after some investigating(and trying the swapoff thing), I found that my /tmp directory was filling up and was at 100%. I grew that directory and it solved it…may look into cleaning up /tmp more frequently if it continues to be a problem.

Hi,

I had exactly the same issue, and I was able to resolve the issue using command below:

export KUBECONFIG=/etc/kubernetes/kubelet.conf

Sometimes it affects master node as well, but in that case - another issue appears, somethings like:

unable to connect to the server: x509: certificate signed by unknown authority

1 Like

I had the same issue every time i reboot the system in my Debian master node so i realized that ‘swapoff -a’ was a volatile command. Then i saw when the system comes up that it truly has a swap partition on, despite i comment this out on /etc/fstab and i disabled it and deleted it. My surprise came when i issue the ‘swapon -s’ and it showed me /dev/sda12 as a swap enabled.
‘systemctl status /dev/sda12’ showed me it up and running. (also i used ‘lsblk -f’)
This partition was outside of my Debian partition so i launched GParted software from my Debian partition (gnome) and delete it. No more problems. :wink:

1 Like

Hi changec,
I’m having a same problem tried above still no use. Can you please help.


Thanks,
BK

1 Like

Hi ,
You can try “systemctl enable docker” inorder to turn on the docker. Before that, you are supposed to reboot/restart the terminal and try the above command.

Thank you! Have a great day! Let me know if it works.

Hi tania,

Thank you for the response. Unfortunately, it’s still the same. Please, find the attached screenshot.

Thanks
BK

Thanks it works for me

Hi there, if you are still having this issue as I did, there are quite a few things that could be wrong. Ensure swap is off definitely as advised by other comments. Other interesting causes from my sujjourn include.

  1. The Kubeconfig environmental variable is probably not set.
    export KUBECONFIG=/etc/kubernetes/admin.conf or $HOME/.kube/config
  2. The user’s $HOME directory has no .kube/config file.
    If you don’t have a .kube or config file
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf HOME/.kube/config sudo chown (id -u):$(id -g) $HOME/.kube/config
    Alternatively you can also export KUBECONFIG variable like this:
    export KUBECONFIG=$HOME/.kube/config
  3. The server/port configured in the config file above is wrong.
    Is it the same as the IP/hostname of the master server? if not did you copy it? You might
    want to fix that.
    By the way you can get the hostname by issueing the hostname command on your cli. or
    ifconfig for the ip. :slight_smile:
  4. Kubelet service may be down. This may be due to the fact that swap is enabled.
    sudo swapoff -a
    To make it permanent go to /etc/fstab
    sudo -i
    swapoff -a
    exit
    strace -eopenat kubectl version
    sudo systemctl restart kubelet.service
  5. Docker service may be down, hence the kubeapi pod isn’t running
    sudo systemctl start docker
    sudo systemctl start kubelet
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown (id -u):(id -g) $HOME/.kube/config

  1. Firewalls may be blocking the access
    sudo systemctl status firewalld #redhat centos
    sudo systemctl stop firewalld #redhat, centos
    sudo ufw status verbose #ubuntu
    sudo ufw disable #ubuntu
5 Likes

check my response below

1 Like

Thanks! That saved me four and a half days of troubleshooting!
Disk space was the cause in my case as well.
When I did “df -h”, no overlay or shm (mounted on /var/lib/docker…) was listed, until I increased the free disk space.

I came here for a similar problem and while your post didnt directly solve my issue, it helped guide me to where my issue was.

In my case on reboot my kubelet service was down, and nothing was starting due to swap being-on even though I disabled it.

You mention how to disable it but it doesn’t work.

I think a lot of people are following the same guides, and the issue is that the guides are all the exact same and contain the same snippet of cause which is causing this problem. These 3 lines need to be run individually, and not strung together:

sudo dphys-swapfile swapoff && \
sudo dphys-swapfile uninstall && \
sudo systemctl disable dphys-swapfile

remove the && and \ and copy and paste each line in the command line, this will ensure swap is disabled and persistent across a reboot.

2 Likes

Please try
strace -eopenat kubectl version
either using sudo or root.

the above command worked for me. i too banged head a lot
i have deployed Kubernetes on Ubuntu16.04 on Virtualbox