Install kubespray fail initializing first master , provide openstack with http_proxy

General Discussions
1

version: 1.14.2
Cloud being used: openstack
Installation method: kubespray | hands on
Host OS: ubuntu 16.04

Hi, is this my first time in here,
so if i’m do some thing wrong, please tell me…

I’m use de kubespray in the last version to install my cluster with openstack provide behind proxy only in production in DEV cluster dont have a proxy and any fail happening to install …
de both configuration is the same, only diff is about the proxy settings
http_proxy and https_proxy conf on inventory/grup_vas/all/all.yaml.

In first place i had many issue with proxy, they ask to pass all domain in use on the installation to conf allow each one than.

i believe this part is solve now.

so to debug where the install broken, i reset the cluster and install hands on without kubespray with kubeadm, the cluster is just one master and one node, to try discovery what going on,

and all was fine until to configure provide openstatck.

I’m kubeadm to init with calico network and
kubadm init --pod-network-cidr=192.168.0.0/16
and
kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml

was ok, all may pods is fine

kube-system   pod/calico-kube-controllers-658558ddf8-sstdp          1/1     Running   0          21h
kube-system   pod/calico-node-nvrzk                                 1/1     Running   0          9h
kube-system   pod/calico-node-sj89v                                 1/1     Running   0          21h
kube-system   pod/coredns-5c98db65d4-fkzwd                          1/1     Running   2          22h
kube-system   pod/coredns-5c98db65d4-fvcsz                          1/1     Running   2          22h
kube-system   pod/etcd-tivit-aiops-k8s-preprod-master-2             1/1     Running   0          22h
kube-system   pod/kube-apiserver-tivit-aiops-k8s-preprod-master-2   1/1     Running   0          22h
kube-system   pod/kube-proxy-ltwt9                                  1/1     Running   0          22h
kube-system   pod/kube-proxy-sg6kw                                  1/1     Running   0          9h
kube-system   pod/kube-scheduler-tivit-aiops-k8s-preprod-master-2   1/1     Running   0          22h
kube-system   pod/tiller-deploy-7bf78cdbf7-wmrmd                    1/1     Running   0          8h

but bem config the cloud provide, kubelet begin show somes erros,

Jul  8 04:50:27 ubuntu kubelet[18915]: F0708 04:50:27.261550   18915 server.go:273] 
failed to run Kubelet: could not init cloud provider "openstack": Post https://api.br-spo 
-1.tivitcloud.com:5000/v3/auth/tokens: dial tcp 200.185.129.137:5000: i/o timeout

with kubespray this error is broken my installation,
i believe the erro on inicialize is because the kubelet has timeout erro
like above log …

but i just discovery that after install without kubespray, and i thing this happening because
the kubelet ignore the proxy configuration.

so i try to configure the proxy manually on /etc/default/kubelet
point on /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

when apply that after three long days , the erro change.
now is:

Jul  8 05:05:19 ubuntu kubelet[20969]: F0708 05:05:19.386381   20969 server.go:273] failed to run Kubelet: error fetching current node name from cloud provider: unexpected status code when reading metadata from http://169.254.169.254/openstack/2012-08-10/meta_data.json: 503 Service Unavailable
Jul  8 05:05:19 ubuntu systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a

i understand it now, hist can find my cloud provider, some thing is wrong yet .

i configure my cloud provide like this way.

on /etc/kubernetes/cloud.conf

[Global]
auth-url="https://mydomain-openstack:5000/v3"
username="user"
password="pass"
tenant-id="my-project-id"
domain-name="domain"

add two line on /etc/kubernetes/manifests/kube-controller-manager.yaml
- --cloud-provider=openstac
- --cloud-config=/etc/kubernetes/cloud.confline

yml
    spec:
      containers:
      - command:
        - kube-controller-manager
        - --use-service-account-credentials=true
        - --controllers=*,bootstrapsigner,tokencleaner
        - --root-ca-file=/etc/kubernetes/pki/ca.crt
        - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
        - --address=127.0.0.1
        - --kubeconfig=/etc/kubernetes/controller-manager.conf
        - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
        - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
        - --leader-elect=true
        - --cloud-provider=openstack
        - --cloud-config=/etc/kubernetes/cloud.conf

in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

i add the same conf
--cloud-provider=openstack --cloud-config=/etc/kubernetes/cloud.conf

Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --cloud-provider=openstack --cloud-config=/etc/kubernetes/cloud.conf"```

restart the daemon and service

sudo systemctl daemon-reload
sudo systemctl restart kubelet

and use command to check it’s fine…
ps xau | grep /usr/bin/kubelet

response of the command is:

root     21647  4.5  4.5 421688 93740 ?        Ssl  15:06   0:08 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --cloud-provider=openstack --cloud-config=/etc/kubernetes/cloud.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0

so a believe is conf work , but still render the erro…

error fetching current node name from cloud provider: unexpected status code when reading metadata from http://169.254.169.254/openstack/2012-08-10/meta_data.json: 503 Service Unavailable

i miss some thing ? how i can troubleshooting to find, if some thing is block by proxy ,

i need configure http_proxy on kubelet ? if yes, this correct place ?

any idea ?

thanks !