Installing behind a proxy

General Discussions microk8s
1

Configuring the proxy environment variables

MicroK8s packages a number of utilities (eg curl, helm) that may need to fetch resources from the internet. Having our environment (/etc/environment) properly configured allows such utilities to function properly. We need to configure HTTPS_PROXY, HTTP_PROXY, https_proxy and http_proxy variables with the proxy endpoint as well as NO_PROXY and no_proxy with the IP ranges the pods and services occupy. For instance assuming our proxy is at http://squid.internal:3128, /etc/environment should set the following environment variables:

HTTPS_PROXY=http://squid.internal:3128
HTTP_PROXY=http://squid.internal:3128
NO_PROXY=10.0.0.0/8,192.168.0.0/16,127.0.0.1,172.16.0.0/16,.svc,localhost
https_proxy=http://squid.internal:3128
http_proxy=http://squid.internal:3128
no_proxy=10.0.0.0/8,192.168.0.0/16,127.0.0.1,172.16.0.0/16,.svc,localhost

Host configuration and clustering behind a proxy

Any nodes where MicroK8s is installed need to be included in the no_proxy environment variable of the host. Usually this can be be done by editing /etc/environment.

DNS configuration

By default the DNS addon (available with microk8s enable dns) points to Google’s 8.8.8.8 and 8.8.4.4 servers for resolving addresses outside the cluster. If these forward DNS endpoints are not reachable you may want to reconfigure CoreDNS as described in the addon’s documentation page.

Configuring containerd only

Containerd uses the proxy configuration set in /etc/environment, therefore no extra configuration is typically required. However, we have the option to set the proxy settings only for containerd. To do so we need to enter the proxy details in

${SNAP_DATA}/args/containerd-env (normally /var/snap/microk8s/current/args/containerd-env). The containerd-env file holds the environment variables containerd runs with. Setting the HTTPS_PROXY to our proxy endpoint enables containerd to fetch conatiner images from the web. We may also want to set the NO_PROXY environment variable with the cluster-cidr and the services-cidr as specified in the respective configuration files /var/snap/microk8s/current/args/kube-proxy and /var/snap/microk8s/current/args/kube-apiserver.

Here is an example where HTTPS_PROXY environment variable is set to http://squid.internal:3128 and NO_PROXY to 10.0.0.0/8,192.168.0.0/16,127.0.0.0/8,172.16.0.0/16,.svc,localhost:

HTTPS_PROXY=http://squid.internal:3128
NO_PROXY=10.0.0.0/8,192.168.0.0/16,127.0.0.1,172.16.0.0/16,.svc,localhost
#
# Some additional environment variables
#
ulimit -n 65536 || true
ulimit -l 16384 || true

For the changes to take effect we need to restart MicroK8s:

sudo snap restart microk8s
2

Some feedback on this page:

  1. Thank you for this page existing! My first attempt at Kubernetes is using a VM on a company server that’s behind a firewall, so I did need to set up a proxy and it was useful to have this page here.

  2. The default of https://squid.internal:3128 seemed appropriate to me but didn’t work for me in the end. I had just uncommented the default in /var/snap/microk8s/current/args/containerd-env. However it turned out that it needed to be http://squid.internal:3128 (not HTTPS). I understand that this should be the common case for most (all?) users who need to configure a proxy; is there anyone who actually uses HTTPS to their own local proxy? If not, then can I suggest that both the doc example and the commented out example in the configuration file be changed?

  3. There was no indication of what the problem was. Perhaps this page could document what the user expects to see when there is a connection problem that might need a proxy configuration? In my case I was following the Getting started page. The microk8s status --wait-ready command hung forever. Running microk8s status told me microk8s is not running. Use microk8s inspect for a deeper inspection. but microk8s inspect listed all services as running. It wasn’t clear to me if this was a proxy configuration problem or something else entirely.

  4. There was no indication of what to do specifically in the Getting started case when proxy configuration is required. Is microk8s stop; microk8s start supposed to fix it? Because I hadn’t yet figured out the configuration error, it didn’t fix it, and I was left wondering if the problem was that some bootstrap step had failed and wouldn’t attempt to run again. In the end, once I worked out the actual problem, microk8s stop; microk8s start did work; my point is some positive affirmation that this is expected to work even on an initial bootstrap failure would have been helpful. Or even better, it’d be nice to be able to supply the proxy configuration prior to any failure, and for the quick start documentation to mention that.

3

Hi Robie! Great to hear from you.

That’s a great point and a fairly easy fix. I will take advice on what the commonest scenario is, but it is easy to add some note or explanation there to call out that your endpoint may be different.

I agree, a lot of the time knowing exactly what the problem is is 90% of solving it. We can add some more troubleshooting help around this. I wonder @kjackal - is there some nicer way MicroK8s itself could surface such issues?

The getting started is really intended for the general use case. I am reluctant to add too many extra’diversions’ in there, but I agree that we should at least signpost this better, and I will see if I can take a fresh look at the user journey here with your comments in mind.

Really, thanks so much for providing your feedback and spending the time to detail your experience - it is so much more useful than “it doesn’t work” comments!

4

Thank you for looking at this!

I understand the need to not overcomplicate the general use case directions, so I leave that to your judgement. If you choose not to do that, I wonder if instead referring to the “Getting started” instructions from the “Installing behind a proxy” page might work instead. Specifically so readers can understand how the two things are to be integrated even if the instructions themselves are on the latter page. That’s the connection that I had missed. I suppose the two specific questions I had there were:

  • Were the proxy instructions intended/capable to fix the Quick start or not?
  • If yes, then at what point in the Quick start instructions was it intended that I intervene to configure the proxy, where should I resume in the Quick start instructions when done, and what (if any) steps needed redoing in the Quick start instructions before resuming?
5

Hi,

Could you also provide an example of " You may also want to set the NO_PROXY to the cluster-cidr and the services-cidr as specified in the respective configuration files /var/snap/microk8s/current/args/kube-proxy and /var/snap/microk8s/current/args/kube-apiserver ." ?

I’ve tried setting it by adding --no-proxy=“10.1.0.0/16,10.152.183.0/24” in both files, and I end up locked out of microk8s. After restarting the snap, I am no longer able to query kubernetes:

$ microk8s.kubectl get all 
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
6

There is no arg --no-proxy on both kube-proxy and apiserver. I think right now it doesn’t start at all.

7

Interesting. Why is the documentation suggesting the following then?

8

What i understood from the docs is that one can add to the NO_PROXY environment variable the value of the cluster-cidr in the kube-proxy file and the value of service-cluster-ip in the kube-apiserver. But not to add a --no-proxy arguments to kube-apiserver and kube-proxy.

1 Like
9

Looking at the files in question, there is a cluster-cidr in /var/snap/microk8s/current/args/kube-proxy, but there is no services-cidr in /var/snap/microk8s/current/args/kube-apiserver.

I’m assume you meant service-cluster-ip-range, since the value from the file:
--service-cluster-ip-range=10.152.183.0/24
matches what you show for NO_PROXY.

Any nodes where MicroK8s is installed on need to be included in the no_proxy environment variable of the host. Usually this can be be done by editing /etc/environment .

Does this include localhost, 127.0.01, or just IPs/names of other hosts?

10

FWIW, the 172.16.0.0 range is actually a /12 block, not a /16, according to RFC 1918 Section 3

1 Like
11

It is worth adding this to your no_proxy and NO_PROXY env when enabling Metallb

no_proxy=10.0.0.0/8,192.168.0.0/16,127.0.0.1,172.16.0.0/16,webhook-service.metallb-system.svc
1 Like