Akito
September 7, 2022, 1:45pm
1
Kubernetes version: v1.23.3
ingress-nginx requires Kubernetes v1.19.0 or higher
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0x1)
k8s.io/klog/v2@v2.9.0/klog.go:1026 +0x8a
k8s.io/klog/v2.(*loggingT).output(0x28a9ac0, 0x3, {0x0, 0x0}, 0xc00048e070, 0x0, {0x1f86a9b, 0x1968996}, 0x1c, 0x0)
k8s.io/klog/v2@v2.9.0/klog.go:975 +0x63d
k8s.io/klog/v2.(*loggingT).printf(0x0, 0x0, {0x0, 0x0}, {0x0, 0x0}, {0x1992456, 0x33}, {0x0, 0x0, ...})
k8s.io/klog/v2@v2.9.0/klog.go:753 +0x1e5
k8s.io/klog/v2.Fatalf(...)
k8s.io/klog/v2@v2.9.0/klog.go:1514
main.main()
k8s.io/ingress-nginx/cmd/nginx/main.go:107 +0x8b2
goroutine 18 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
k8s.io/klog/v2@v2.9.0/klog.go:1169 +0x6a
created by k8s.io/klog/v2.init.0
k8s.io/klog/v2@v2.9.0/klog.go:420 +0xfb
Which should be coming from here.
func NetworkingIngressAvailable(client clientset.Interface) bool {
version119, _ := version.ParseGeneric("v1.19.0")
serverVersion, err := client.Discovery().ServerVersion()
if err != nil {
return false
}
runningVersion, err := version.ParseGeneric(serverVersion.String())
if err != nil {
klog.ErrorS(err, "unexpected error parsing running Kubernetes version")
return false
}
return runningVersion.AtLeast(version119)
}
The thing is, this started to happen without any reason or trigger in the middle of normal operation.
This Ingress Controller has been working fine for over 200 days straight. Now I am getting this error for no reason, at all.
How can this be permanently fixed?
Theog75
September 7, 2022, 5:20pm
2
did something change on your cluster? kubelet versions? did you upgrade your ingress controller?
Akito
September 7, 2022, 5:29pm
3
Literally didn’t even look at it for weeks. Just looked at it today, because I suddenly wasn’t able to access my online services provided through that cluster.
Theog75
September 7, 2022, 6:19pm
4
can you run
kubelet --version
on your nodes and send the output?
Akito
September 7, 2022, 6:32pm
5
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
It’s a simple single-node Cluster. Nothing out of the ordinary. Pretty standard, no extra tools to manage Kubernetes or anything like that. Just plain old Kubernetes.
Theog75
September 7, 2022, 10:47pm
6
This is the kubectl version out not kubelet
Akito
September 9, 2022, 12:04am
8
“Solved” it by applying the following fix.
What an unnecessary and random pain.
Will this happen randomly at some point again? I don’t know.
Just to emphasize, that this was absolutely uncalled for:
I did not change anything on the cluster. This issue literally appeared randomly out of thin air.
1 Like