If you add a PDB for it, that should help:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: kube-dns-autoscaler-pdb
namespace: kube-system
labels:
app: kube-dns-autoscaler
spec:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns-autoscaler
(You might do the same thing for kube-dns for the same reason.)
Dan