Cluster information:
Kubernetes version: 1.32.3
Installation method: Binary deployment
Host OS: ubuntu22.04
CNI and version: Calico v3.29.3
CRI and version: containerd://2.0.4
haproxy/VIP：192.168.110.208【k8s-lb-vip.k8s.cluster】
一、Deploy metrics-server 【Normal use】
metrics installed fine, and kubectl top node should output it
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml -O metrics-server-ha.yaml
kubectl apply -f metrics-server-ha.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: “true”
rbac.authorization.k8s.io/aggregate-to-edit: “true”
rbac.authorization.k8s.io/aggregate-to-view: “true”
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources: - pods
- nodes
verbs: - get
- list
- watch
- metrics.k8s.io
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- “”
resources: - nodes/metrics
verbs: - get
- “”
- apiGroups:
- “”
resources: - pods
- nodes
verbs: - get
- list
- watch
- “”
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
replicas: 2
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
k8s-app: metrics-server
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
k8s-app: metrics-server
namespaces:
- kube-system
topologyKey: kubernetes.io/hostname
containers:
- args:
- --cert-dir=/tmp
- --secure-port=10250
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --v=2
- --kubelet-insecure-tls
- --requestheader-client-ca-file=/ssl/demoCA/newcerts/k8s-ca.crt
image: k8s.m.daocloud.io/metrics-server/metrics-server:v0.7.2
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 10250
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
initialDelaySeconds: 20
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 200Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /tmp
name: tmp-dir
- name: ca-certificates
mountPath: /ssl
readOnly: true
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
- name: ca-certificates
hostPath:
path: /ssl
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
minAvailable: 1
selector:
matchLabels:
k8s-app: metrics-server
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
二、Deploy prometheus adapter
wget -P /download/ https://github.com/prometheus-community/helm-charts/releases/download/prometheus-adapter-4.14.1/prometheus-adapter-4.14.1.tgz
cd /download
tar -zxvf prometheus-adapter-4.14.1.tgz
cd /download/prometheus-adapter
helm template myrelease /download/prometheus-adapter --output-dir /download/prometheus-adapter/output
ls /download/prometheus-adapter/output/prometheus-adapter/templates
cluster-role-binding-auth-delegator.yaml configmap.yaml custom-metrics-cluster-role.yaml service.yaml
cluster-role-binding-resource-reader.yaml custom-metrics-apiservice.yaml deployment.yaml serviceaccount.yaml
cluster-role-resource-reader.yaml custom-metrics-cluster-role-binding-hpa.yaml role-binding-auth-reader.yaml
I modified the deployment.yam file as follows:
args:
- –prometheus-url=http://192.168.110.227:9090
*PS：All other .yaml is the default and has not been changed
kubectl create -f /download/prometheus-adapter/output/prometheus-adapter/templates/
Displaying 404 error
Message: failing or missing response from https://10.244.134.39:6443/apis/custom.metrics.k8s.io/v1: bad status from https://10.244.134.39:6443/apis/custom.metrics.k8s.io/v1: 404
kubectl describe apiservices v1.custom.metrics.k8s.io
Name: v1.custom.metrics.k8s.io
Namespace:
Labels: app.kubernetes.io/component=metrics
app.kubernetes.io/instance=myrelease
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=prometheus-adapter
app.kubernetes.io/part-of=prometheus-adapter
app.kubernetes.io/version=v0.12.0
helm.sh/chart=prometheus-adapter-4.14.1
Annotations:
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2025-04-10T06:54:15Z
Resource Version: 1104493
UID: 82a33cfb-118f-436b-8a6c-da87a55c0ca7
Spec:
Group: custom.metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: myrelease-prometheus-adapter
Namespace: monitoring
Port: 443
Version: v1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2025-04-10T06:54:15Z
Message: failing or missing response from https://10.244.134.39:6443/apis/custom.metrics.k8s.io/v1: bad status from https://10.244.134.39:6443/apis/custom.metrics.k8s.io/v1: 404
Reason: FailedDiscoveryCheck
Status: False
Type: Available
Events:
journalctl -f -u kube-apiserver
Apr 10 16:20:34 k8s-master-01 kube-apiserver[1389104]: E0410 16:20:34.252015 1389104 remote_available_controller.go:448] “Unhandled Error” err=“v1.custom.metrics.k8s.io failed with: failing or missing response from https://10.244.134.39:6443/apis/custom.metrics.k8s.io/v1: bad status from https://10.244.134.39:6443/apis/custom.metrics.k8s.io/v1: 404” logger=“UnhandledError”
Apr 10 16:20:35 k8s-master-01 kube-apiserver[1389104]: I0410 16:20:35.834092 1389104 apf_controller.go:493] “Update CurrentCL” plName=“exempt” seatDemandHighWatermark=1 seatDemandAvg=0.0065918790569620325 seatDemandStdev=0.08092234665072694 seatDemandSmoothed=0.10054696971198927 fairFrac=2.330357142857143 currentCL=1 concurrencyDenominator=1 backstop=false
Apr 10 16:20:42 k8s-master-01 kube-apiserver[1389104]: E0410 16:20:42.752875 1389104 remote_available_controller.go:448] “Unhandled Error” err=“v1.custom.metrics.k8s.io failed with: failing or missing response from https://10.244.134.39:6443/apis/custom.metrics.k8s.io/v1: bad status from https://10.244.134.39:6443/apis/custom.metrics.k8s.io/v1: 404” logger=“UnhandledError”
What is the reason, please help me see, thank you