Microk8s 1.17.3 to 1.17.4: "x509: certificate signed by unknown authority"

General Discussions
1

Cluster information:

Kubernetes version: microk8s 1.17.4
Cloud being used: bare-metal
Installation method: snap
Host OS: entOS Linux release 7.7.1908 (Core)

Question: What changed from 1.17.3 to 1.17.4 that caused x509 errors? How can I fix them (behind a zscaler firewall), or get 1.17.3 installed, before I pull all my hair out?

Background:

On a dev VM server where microk8s (basic quick install with dns) was working well with an rstudio installation until snap upgraded me from 1.17.3 to 1.17.4 and then things stopped working. I was getting “x509: certificate signed by unknown authority”. In the end I did a snap revert to 1.17.3 and it worked ok again, all ok on the dev server. I am behind a fancy zscaler proxy/filter btw. I thought these point releases were not supposed to break anything? They did - or seemed to anyway.

Now I want to install this all on a different bare-metal prod host but of course snap has no way to install 1.17.3 and again using 1.17.4 immediately gives me an x509 headache even before I have set up rstudio, just doing a basic test pod install the image pull from k8s.gcr.io fails with flaming x509. Of course I can’t revert to 1.17.3 because it was never installed on this host, thanks snap.

I think this is a common problem and it would be nice if there was some guidance on this in the microk8s setup. I can’t set an HTTPS_PROXY because zscaler doesn’t work like that. I’ve tried setting insecure-skip-tls-verify: true which doesn’t help. I’m no x509 or kubernetes wiz and need a straightforward way to get this working - any ideas?

Many thanks for your help,

Tim

2

I’m not sure this is actually a zscaler issue, it seems to be something to do with storage.googleapis.com (which is used by the pause container)? Why does the issue only occur after auto-upgrade of microk8s
to 1.17.4 (and goes away again after a revert to 1.17.3)? Any way to solve it in 1.17.4? Or some way to install a snap with 1.17.3.

There should be away in snap to install a reverted snap from one machine to another, since otherwise you’re forced on the new machine to install the snap version that is broken for you. I’m not entirely sure if you can use snap save on one machine then export and install it to another machine to achieve this?

3

Anyone else have this x509 issue behind firewalls?